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ABSTRACT 

Development and validation of the Texas Oral 
Proficiency Tesc (TOPT) is described. The TOPT, a simulated oral 
proficiency interview, was developed in French and Spanish as a test 
of speech skills to be used by the state as part of its certification 
testing program for French, Spanish, and bilingual education 
teachers. An introductory chapter provides background information, 
beginning with the history leading to the test's development and 
concluding with a description of the test itself. The second chapter 
describes a survey to establish the job relevance of the test in each 
of the three teaching areas. The third chapter chronicles the 
development of the trial form of the TOPT. Chapter four describes the 
trial process, including the purpose, recruiting examinees, 
adminstration and data collection, and the results of examinee data 
forms and judge sheets. Development of the final form is outlined in 
chapter five, and content validation is addressed in the sixth 
chapter. The final chapter describes how passing score standards for 
the TOPT were set through three separate studies. Appended materials, 
which form the bulk of the document, include the survey instruments, 
response, evaluation, rating forms, data on trialing examinees, item 
ratings and sample comments, and related documents. (MSE) 
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1. Introduction 

This report describes the development and validation of the 
Texas Oral Proficiency Test (TOPT) . The TOPT was developed by 
the Center for Applied Linguistics under contract with the Texas 
Education Agency (TEA) and with the cooperation of numerous 
French, Spanish and bilingual education specialists from 
throughout the state of Texas. The TOPT was developed in French 
and Spanish as a test of speaking proficiency, to be used by the 
state as a part of its teacher certification testing program for 
persons seeking certification or endorsements in French, Spanish 
or bilingual education. 

This introductory chapter gives the background to the 
development of the Texas Oral Proficiency Test (TOPT). It begins 
with a description of the history leading up to its development 
and concludes with a description of the test itself. 

1.1 Background 

In 1981 the Texas legislature passed Senate Bill 50, which 
requires that persons seeking certification as a K-12 educator in 
Texas public schools perform satisfactorily on comprehensive 
examinations. Senate Bill 50 also mandated the establishment of 
the Commission on Standards for the Teaching Profession (CSTP) , 
which was created in 1982, to oversee teacher education 
standards. The State Board of Education (SBOE) mandated the 
development of tests as part of the state's teacher certification 
requirements. The purpose of these tests is to ensure that each 



teacher has the necessary knowledge to teach in Texas public 
schools. 

Since May 1986, all persons applying for teacher education 
in Texas have been required to pass the Examination for the 
Certification of Educators in Texas (ExCET) . ExCET examinees 
must take either a general elementary or secondary education 
test. Those applying for secondary certification must pass a 
test on knowledge of the content of their area of specialization. 
For language educators, content specialization tests are 
available in Spanish, French, German, Latin, Bilingual Education 
and English as a Second Language. These tests assess a variety 
of knowledge, including both the ability to comprehend written 
material and the ability to ^onstrate knowledge of the language 
acquisition process, language teaching methodology, contrast ive 
linguistics, and so forth. 

With the exception of bilingual education teachers, language 
educators have not been required to pass a test that requires 
them to demonstrate their ability to speak the language. This 
situation exists despite Hou«e Bill 246 of 1982, which 
established foreign language curricula based on the development 
of proficiency among foreign language students. Thus, although 
Texas educators h^ve been required to develop proficiency 
(including oral language proficiency) in their students for 
almost a decade, new foreign language teachers have not been 
required to pass an examination that assesses their own oral 
language proficiency. 



The Texas SBOE, at its January 1986 meeting, recommended 
that future French, German, and Spanish teachers be assessed 
using procedures, criteria, and a passing score in accordance 
with the Language Proficiency Guidelines developed by the 
American Council for the Teaching of Foreign Languages (ACTFL) . 
Since this decision would have had an impact on Texas' 1984 
Standards for Teacher Education, the SBOE then referred the whole 
matter to the Commission on Standards for the Teaching Profession 
(CSTP). At its March and May 1986 meetings, this committee 
discussed and passed a revision of the Standards for Teacher 
Education which echoed the position £ the SBOE. 

During subsequent years, the Texas Education Agency has been 
struggling to determine a method to implement this requirement 
for foreign language teachers in a way that would ensure 
reliability and validity of measurement. 

Bilingual educators in Texas, however, have been required to 
demonstrate oral proficiency in a second language for some time. 
In 1978, the bilingual education unit at the TEA included the 
training of oral proficiency interviewers in its Title VII 
proposal to the U.S. Department of Education. Utilizing these 
funds, the TEA contracted with Educational Testing Service to set 
up and administer the program locally. In November 1978, 100 
Spanish interviewers from colleges and universities throughout 
the state were trained in a one-day training session. Fifteen 
raters from Texas' regional educational service centers were also 
trained and eight of these were assigned the duty of rating 



interviews as part of their job responsibilities. In subsequent 
years , as interviewers and raters lost interest in the program, 
additional personnel were trained by ETS under contract with the 
TEA. In tine the program evolved so that the ratings were also 
done by professors at universities located in a different part of 
the state from where the interview was held. 

Because the ACTFL Guidelines did not exist at the time, 
Texas began using what was then called the Foreign Service 
Institute oral proficiency scale, which runs from 0 to 5. Texas 
began referring to the test as the Language Proficiency Interview 
(LPI) , and established a minimum rating of 3 on the LPI for 
certification. Although applicants could initially take their 
first LPI without charge, eventually the policy was changed so 
that all applicants paid $45 for each interview. Up to the 
writing of this report, this program continues to be managed by 
the ETS Austin Field Service Office. 

While the LPI program has continued to function for over a 
decade, it has suffered from a number of problems. First, there 
• is the difficulty of matching interviewers with examinees 

throughout the state. The process can be inconvenient for both 
the interviewer and the examinee. There are also the standard 
problems of reliability and validity. In the LPI, these were 
reflected in both the interviews and the ratings. Often, raters 
would indicate that an interview was not competently done; that 
is, that it did not produce a ratable sample of speech. In 
addition, interviewers, upon learning the rating assigned by a 
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rater in another part of the state, sometimes disagreed with the 
\ rating assigned. Similarly, professors who were familiar with 

* the LPI scale and the proficiency level of their students, also 

frequently disae/reed with the rating assigned. The most frequent 
complaint was that the ratings were too generous; that is, that 
many students were given a 3 when their true proficiency was 
level 2 or 2+. 

The Division of Teacher Assessment of the TEA, as well as 
the bilingual education and foreign language specialists at the 
TEA, were aware of these problems with the LPI. Not wishing to 
either continue the current situation for bilingual educators or 
to extend it to foreign language educators, they issued a Request 
for Proposals to develop a new testing program on January 9, 
1990. The RFP called for the development of Spanish and French 
tests to be used to certify Spanish, French, and Bilingual 
Education teachers. The RFP did not specify the format of the 
test to be developed, thus either a face-to-face test or a semi- 
direct test could be proposed. 
~ The Division of Foreign Language Education and Testing of 

the Center for Applied Linguistics proposed to utilize the 
Simulated Oral Proficiency Interview (SOPI) format which it had 
designed and had already applied very successfully to the 
development of semi-direct tests in five languages. The SOPI 
format has been shown to correlate as highly with the OPI as the 
OPI correlates with itself. In addition, it offers greater 
standardization and control, which is important in a large scale 
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testing program, and especially important with high stakes tests 
such as those used to certify teachers. Parallel forms of the 
SOPI can be developed to alleviate the concern about security 
that occurs when only a single form is available. Yet, parallel 
forms of the SOPI, unlike different interviewers, can be 
developed under strict guidelines and subsequently pilot tested 
and revised to ensure that the forms are comparable. 

The rating of a SOPI is facilitated by the fact that all 
examinees take the same test. Under these circumstances, it is 
easier to place examinees on the ACTFL scale. To illustrate how 
a SOPI facilitates reliable rating, a parallel can be drawn with 
the scoring of essays. Using any given scale, a packet of essays 
on the same topic vill normally be rated more reliably than a 
packet of essays on different topics. The SOPI is invariant 
across examinees at the same administration, while the OPI varies 
across examinees at the same administration. 

Because the SOPI seemed to offer significant controls over 
reliability and validity, CAL was awarded a contract to develop 
three forms of a SOPI in Spanish and three forms in French for 
Texas educators. The contract was awarded on April 1, 1990, and 
work began immediately. Because these particular SOPIs would 
become the property of the TEA, and because they were designed 
for a particular population (educators) , it was decided to give 
them a different name. Eventually, the name Texas Oral 
Proficiency Test (TOPT) was chosen. 

When implemented in the fall of 1991, the TOPT will replace 
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the Language Proficiency Interview (LPI) that has been required 
for the bilingual education endorsement. All persons seeking 
either an endorsement or a certificate in bilingual education 
must pass the TOFT. 

A description of the TOFT follows in this chapter, while the 
remaining chapters describe the development and validation of the 
TOPT in detail. 

1.2 Descripti on of the TOPT 

The TOFT is a semi-direct, tape-mediated test of oral 
language proficiency that is taken in a language lab. The 
examinee hears the directions and items for all parts of the test 
from a master test tape. In addition, in three of the four parts 
of the test, the examinee uses pictures and other information 
from a test booklet to answer items. All responses are recorded 
on a separate examinee response tape. 

Because the TOPT is a test of speaking ability (not 
listening comprehension), the general directions to the test and 
the directions for each item are in English. However, each item 
ends with a target language question or statement heard on the 
master tape. Following the English directions and in response to 
this target language prompt, all examinee responses are spoken in 
French or Spanish into the microphone and recorded on the 
response tape. 

Once the master tape begins, the test cannot be stopped. 
The master test sets the pace of the test and lasts approximately 
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45 minutes. The examinee speaks French or Spanish for 
approximately 20 minutes during timed pauses throughout the test. 
The examinee response tape is subsequently evaluated by trained 
raters approximately two weeks following the examination. 

The TOPT consists of a warm-up section followed by fifteen 
items designed to allow the examinee to demonstrate the ability 
to perform a variety of speaking tasks covering a variety of 
topics and a variety of situations. All the directions are given 
in English. After the examinee hears the directions for each 
item, he or she is given time to prepare the response, usually 
between 15 and 30 seconds. Then, after hearing a statement or 
question in French or Spanish, the examinee responds in the time 
allowed. Again, all responses are recorded on the examinee 
response tape. The following sections describe the TOPT in more 
detail. 

Warm-up 

A warm-up follows the reading of the general directions. 
This section is designed to put the examinee at ease, to allow 
the examinee to make the transition to speaking in the target 
language, and to become comfortable with the test situation. In 
the warm-up, a native speaker of Spanish or French asks the 
examinee several personal background questions, involving his or 
her educational background, interest in teaching and experience 
with the language. The warm-up items are psychometrically 
appropriate for examinees at the Intermediate levels, and would 
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be considered quite easy by examinees at higher levels. 

Picture-Based Items 

A set of five picture-based items follow the warm-up. The 
first requires the examinee to give directions according to a 
route identified on a pictorial nap. The second calls for a 
description of the objects and activities taking place in a 
picture depicting a familiar setting, such as a home or school. 
The third picture requires the examinee to describe a typical set 
of routine events depicted in the pictures. The fourth calls for 
the examinee to retell an amusing event that happened in the 
past. The events and sequence of this story are also clearly 
depicted by pictures. The fifth and final picture-based item 
calls for a description of an event that is planned for the 
future. This event is also presented by a series of pictures. 
The picture-based items are designed to permit the examinee to 
demonstrate the ability to organize discourse in a way that would 
permit him to describe a place, to give directions, and to 
narrate events in present, past, and future time. The picture- 
based items are psychometrically most appropriate for examinees 
at the Intermediate High and Advanced levels. Examinees at 
higher levels would find these speaking tasks to be fairly easy 
to perform, while examinees at lower levels may experience some 
difficulty producing the kind of connected discourse that these 
items require. 
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Topic Items 

The next set of five items allows the examinee to 
demonstrate the ability to speak about a variety of topics. The 
examinee is asked to present advantages and disadvantages of a 
certain proposition, such as using public transportation, to give 
someone step by step directions on how to do something, to 
present a brief factual summary on a familiar topic, such as 
current events or matters pertaining to the state of Texas, or to 
present and support an opinion on a topic related to society or 
education. 

Topic items are generally psychometrically appropriate lor 
examinees at the Advanced and Superior levels. This is because 
they require the examinee to perform speaking tasks that are 
indicative of the kind of language skills that examinees at these 
levels are expected *:o have. Topics involving formal speech, 
such as to a group, or a professional discussion, are appropriate 
for the Superior level examinee. While examinees at lower levels 
will be able to respond to each item, the linguistic and 
rhetorical characteristics of their performance will illustrate 
the limitations in their speaking ability. 

Situation Items 

The final set of five items allows the examinee to 
demonstrate the ability to respond to real-life situations in 
French or Spanish. The examinee may be asked to give advice to a 
friend, to apologize to someone, to lodge a complaint, to resolve 
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a problem, or to attempt to convince someone to take a different 
course of action. These all require the ability to tailor one's 
speech to the individuals and the circumstances presented in the 
item. Situation items on the TOPT, like Topic items, are also 
generally appropriate for examinees at the Advanced and Superior 
levels, although some items designed for the Intermediate level 
examinee may be included. Items designed for higher level 
examinees generally allow higher level examinees to demonstrate 
the range of their linguistic abilities more consistently than do 
items designed for lover level examinees. 
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2. Job-Relatedness Survey 
This chapter describes how the job-relatedness survey for 
the TOFT vas conducted. It presents how the survey questionnaire 
was developed and distributed, and how responses were analyzed. 
This chapter concludes with a presentation of the results of the 
survey. 

2.1 Preparing the Survey 

Because the TOPT is a Simulated Oral Proficiency Interview 
(SOPI) , the Speaking Proficiency Guidelines of the American 
Council for the Teaching of Foreign Languages (ACTFL) and the 
Interagency Language Roundtable (ILR) of the U.S. government lie 
at the heart of the test. As a SOPI, the TOPT consists of a 
series of individual speaking tasks drawn from these Guidelines. 
The TOPT job-relatedness survey (hereafter called the survey) 
used the Guidelines as a point of departure in an effort to 
acquire information on the level of language ability that 
teachers of French, Spanish, and bilingual education in Texas 
need to perform their jobs competently. 

In order to ensure that the speaking tasks to be included 
on the TOPT were appropriate for the population of examinees for 
which the TOPT was intended (prospective Texas classroom 
teachers) , it was necessary to conduct the job-relatedness 
survey. To do this, CAL staff developed a list of 36 speaking 
tasks based on the ACTFL Guidelines. This list of speaking tasks 
was presented for review at the first joint meeting of the French 
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and Spanish TOPT Test Advisory Committees (TACs) at their initial 
meeting on April 4, 1990 (see Chapter Three). Following the 
committee members' revisions and suggestions, a number of tasks 
were further clarified and two more tasks were added, to bring 
the total number of tasks appearing on the questionnaire to 38. 
The speaking tasks ranged in ability level from Intermediate Low 
on the ACTFL scale (e.g., ••Introduce Yourself") to Superior 
(e.g., "Explain a Complex Process in Detail"). "Introduce 
Yourself" was placed first on the list of tasks, while the rest 
of the tasks were presented in random order. Thus, less 
demanding and more demanding speaking tasks were dispersed 
throughout the list appearing in the survey questionnaire. 
Examples of the speaking tasks can be found in the copies of the 
final survey included in Appendix A. 

Instructions for the recipients of the survey questionnaire 
were drawn up jointly by CAL and TEA staff. The task of the 
respondents was to indicate, on a scale of 1 (E) to 5 (A) , 
whether the level of ability required to perform each speaking 
task is needed by French, Spanish or bilingual education teachers 
in Texas, in other words respondents were asked if they believed 
teachers of either French, Spanish or bilingual education should 
possess the level of ability to perform each specified task. 
Respondents indicated their answers by marking the appropriate 
cclumn on a machine-readable respond sheet. Their choices were: 
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A - Definitely Yes 
B - Probably Yes 
C - Maybe 
D « Probably No 
E « Definitely No 

Since the outcome of the TOPT is a score (or level* 
assignment) based on a broad concept of speaking proficiency and 
not only on speaking tasks that may be used in the classroom, the 
speaking tasks presented on the questionnaire were generic. In 
other words, the list included some tasks that teachers do not 
necessarily need to do in the classroom. However, all of the 
tasks represent various levels of ability. In the instructions 
accompanying the survey, each speaking task was described in a 
short paragraph to aid understanding. 

Respondents were also requested to provide certain basic 
demographic data. The questionnaires sent to the three groups of 
teachers were identical except for direct references to group 
membership. Copies of the final survey instructions and machine- 
readable response sheet are included in Appendix A. 

2.2 Distributing the Survey 

A randor sample of 700 Texas classroom teachers was selected 
to receive the survey: 400 in bilingual education, 200 in Spanish 
language teaching, and 100 in French language teaching. For 
bilingual education and Spanish language, these figures represent 
approximately 6 and 8% of the total number of Texas teachers in 
those fields. For French, the figure represents approximately 
10% of the total of that language. 
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The TEA prepared nailing labels for each teacher and the 
principal at each teacher's school. Cover letters to the 
teachers explaining the survey and requesting participation in it 
were also prepared by the TEA. In addition, cover letters 
explaining the importance of the survey to each teacher's 
principal were prepared. All these materials were sent by the 
TEA to CAL. CAL staff then prepared two envelopes: the outer 
one contained the letter to the principal requesting that the 
inner one(s) be hand delivered to the teacher (s) addressed. 
These were mailed out from CAL on April 20, 1990. Response was 
requested no later than May 4, 1990. All responses received by 
May 24, 1990 were included in the tally. Pre-addressed, stamped 
return envelopes were also included in the packet each teacher 
received. 

2.3 Survey Results 

The results of each survey will be presented separately for 
each group. 

2.3.1 Survey Re sults - French Group 

Of the 100 questionnaires sent out to French teachers, 62 
were returned to CAL, for a response rate of 62%. Of these, 1 
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was returned incomplete* and 1 was returned after May 24. The 
following two tables are based on the responses of 60 completed 
questionnaires. Table 2.1 gives the demographic statistics of 
those whose responses could be tallied. 



Table 2.1 

TOPT Survey Result — FRENCH LANGUAGE 
Demographic Statistics of Respondents 



A. Curre nt Level of Assignment 

Level of Assignment Frequency 



Cumulative Cumulative 
Percent Frequency Percent 



Elementary 0 

Jun High/Mid School 8 

High School 49 
(F^ quency Missing « 3) 



0.0 
14.0 
86.0 



0 
8 

57 



0.0 
14.0 
100. 0 



B. Certif ication Held 
Certificate Frequency 



Cumulative Cumulative 
Percent Frequency Percent 



Elem Crt/Frnch Spc 
Second. French Crt 
Both of the Above 
None of the Above 
(Frequency Missing 



1 

53 
1 
2 



3) 



1.8 
93.0 
1.8 
3.5 



1 
54 

55 
57 



1.8 
94.7 
96.5 
100.0 



C. Years of Experience 

Cumulative Cumulative 
Experience Frequency Percent Frequency Percent 



1-2 years 11 

3-5 years 8 

6-10 years 14 

11-15 years 11 

16-19 years 8 

20 or more years 4 
(Frequency Missing = 4) 



19.6 
14.3 
25.0 
19.6 
14.3 
7.1 



11 
19 
33 
44 

52 
56 



19.6 
33.9 
58.9 
78.6 
92.9 
100.0 



1 Several survey questionnaires were returned incomplete. 
These were generally for one of the following three reasons: the 
teacher addressed was r> r longer at the school, the teacher 
addressed was actually in a different teaching field, or the 
address of the school was incorrect. 
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p, Clftffff Lev^g Vwmt in Eft A Thre* ^g ^^^ cumulative 
Class Level Frequency Percent Frequency ^cent 

Ill^Inning 30 52.6 30 52.6 

Most Begin/Some Adv 15 26.3 45 

Half Begin/Half Adv 8 14.0 53 93.0 

Most Adv/Soae Begin 3 5.3 56 • 

All Advanced 1 !•» 57 
(Frequency Missing - 3) 

F„ W l nhfiit Degree Held cumulative Cumulative 

Degree Held Frequency Percent Frequency Ztl---- 

~Bachelor's 30 52.6 30 52.6 

Master's 26 45.6 56 98.2 

oSctoratl 1 1-8 57 100.0 

(Frequency Missing « 3) 

F, EthnicitY Cumulative Cumulative 
Ethnic Group Frequency Percent Frequency ^ff®"* 

Hispanic 5 8.9 5 8.9 

Black 1 1-8 6 10.7 

White 50 89.3 56 100.0 
(Frequency Missing ■ 4) 

G. Sex 

Cumulative Cumulative 

Sex Frequency Percent Frequency Pe f c f"^ 

Male 7 12.3 7 12.3 

Female 50 87.7 57 100.0 

(Frequency Missing - 3) 



Table 2.1 indicates that the typical respondent was a white 
female with a bachelor* s degree and 6-10 years of teaching 
experience. She is certified in secondary level French language 
teaching and teaches all beginning classes at a Texas high 
school. It is interesting to note- that 21.1% had half or more 
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advanced level classes and 47.4% had an advanced degree. 

Table 2.2 presents the mean rating received for earn 
speaking task. The tasks are ordered by average mean ranking. 
The standard deviation is presented in the second column as an 
indication of the agreement or disagreement of the group on the 
mean ranking. The third column presents the approximate ACTFL 
level (l=Intermediate, A«Advanced, S=Superior) of the speaking 
task. The final column shows the speaking task. The line drawn 
indicates the cut-off level of 3.50 below which the speaking 
tasks were not validated for inclusion on the TOPT. 
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Table 2.2 

Results of the Job-Relatedness Survey 

for French 

Std. ACTFL 
Mean Dev. Level Speaking Task 

Introduce Yourself 
Talk About Family Members 
Order a Meal 

Describe Your Daily Routine 
Make Purchases 
Describe Typical Routines 
Give Directions 

Talk About Personal Activities 
Give Instructions 

Describe a Sequence of Events in the Past 
Express Personal Apologies 
Describe a Place 

Explain a Familiar, Simple Process 
Give a Brief Personal History 
Describe Habitual Action in the Past 
Describe Expected Future Events 
Compare and Contrast Two Objects or Places 
Make Arrangements for Future Activities 
Talk About Your Future Plans 
Give a Brief, Organized Factual Summary 
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Table 2.2 indicates that only two Superior level tasks were 
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validated by the French teachers. It also shows that these 
teachers generally ordered these tasks in a way very similar to 
the ACTFL scale. 

2.3.2 Survey Results - S panish Group 

Of the 200 surveys sent to Spanish language teachers, 
121 were returned to CAL for a response rate of 60.5%. Of these 
7 were returned incomplete and 1 was returned after May 24. 
Table 2.3 gives the demographic statistics of the 113 Spanish 
language teachers whose responses could be tallied. 

Table 2.3 

Demographic Statistics of the Respondents 
to the Job-Relatedness Survey (Spanish Language) 



Current Level of Assignment 



Level of Assignment Frequency 



Elementary 15 

Jun High/ Mid School 22 

High School 69 

Other l 
(Frequency Missing « 6) 

B. Certificate Held 
Certificate Frequency 



Elem Cert/Span Spc 12 

Secondary/Span Crt 77 

Both of the Above 7 

None of the Above 10 
(Frequency Missing - 7) 



Cumulative Cumulative 
Percent Frequency Percent 



14.0 15 14.0 

20.6 37 34.6 

64.5 106 99.1 

0.9 107 100.0 



Cumulative Cumulative 
Percent Frequency Percent 



11.3 12 11.3 

72.6 89 84.0 

6.6 96 90.6 

9.4 106 100.0 
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£* Years of Teaching Experience 

Cumulative Cumulative 
Experience Frequency Percent Frequency Percent 



1-2 years 14 13.1 14 13.1 

3-5 years 26 24.3 40 37.4 

6-10 years 26 24.3 66 61.7 

11-15 years 18 16.8 84 78.5 

16-19 years 11 10.3 95 88.8 

20 or more 12 11.2 107 100.0 
(Frequency Missing - 6) 



1L — Levels of Classes Taught in Past Three Years 

Cumulative Cumulative 
Class Level Frequency Percent Frequency Percent 



All Beginning 65 61.3 65 61.3 

Most Begin/Some Adv 20 18.9 85 80.2 

Half Begin/Half Adv 12 11.3 97 91.5 

Most Adv/Some Begin 7 6.6 104 98.1 

All Advanced 2 1.9 106 100.0 
(Frequency Missing » 7) 



Highest Degree Held 

Cumulative Cumulative 
Degree Held Frequency Percent Frequency Percent 



No Degree 1 0.9 1 0.9 

Bachelor's 70 65.4 71 66.4 

Master's 36 33.6 107 100.0 
(Frequency Missing « 6) 



F. Ethnicity 

Ethnic Group Frequency 



Hispanic 46 
Black 3 
White 56 
Other 2 
(Frequency Missing ■ 6) 



"Cumulative Cumulative 
Percent Frequency Percent 



43.0 46 43.0 

2.8 49 45.8 
52.3 105 98.1 

1.9 107 100.0 



Cumulative Cumulative 
Sex Frequency Percent Frequency Percent 



Male 19 17.9 19 17.9 

Female 87 82.1 106 100.0 

(Frequency Missing - 7) 



Table 2.3 indicates that the typical respondent to the 
questionnaire was a white female with a bachelor's degree and 6- 
10 years of teaching experience. She is certified in secondary 
level Spanish language teaching and teaches all beginning level 
courses at a Texas high school. It is worth noting that quite a 
high percentage (43%) of the respondents were Hispanic. A very 
similar percentage to French had half or more advanced level 
classes (19.8% for Spanish versus 21.1% for French). Unlike 
French, however, slightly over one third of the respondents 
(35.5%) were currently assigned to a school other than a high 
school, while that figure was only 14% for French. 

Table 2.4 presents the mean rating for each of the speaking 
tasks presented on the survey. As in the French results, the 
tasks are ordered by average mean ranking. The standard 
deviation is presented in the second column as an indication of 
the agreement or disagreement of the group on the mean ranking. 
The third column presents the approximate ACTFL level 
(I=Intermediate, A=Advanced, S=Superior) of the speaking task. 
The final column shows the speaking task. The line drawn 
indicates the cut-off level of 3.50 below which the speaking 
tasks were not validated for inclusion on the TOPT. 
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The results in Table 2.4 indicate that the Spanish language 
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teachers validated all of the Intermediate and Advanced level 
speaking tasks, but failed to validate 7 of the 12 Superior level 
tasks. As with the French survey, there is a general ordering of 
average responses according to the ACTFL guidelines. However, 
there is a noticeable increase of tasks that are related to the 
classroom but ranked higher on the ACTFL scale, such as "Giving 
Instructions' 1 (ranked second) and "Explain a Simple, Familiar 
Process" which was ranked higher by the teachers than 
Intermediate level items such as "Order a Meal," "Describe a 
Place," "Make Purchases," and "Talk About Personal Activities." 
This may indicate that some of the respondents marked their 
responses according to what speaking tasks they actually used in 
the classroom rather than according to what inherent level of 
ability is required to perform the speaking tasks. 

2.3.1 Survey Results - Bilingual Education Group 

Of the 400 surveys sent to bilingual education 
teachers, 240 were returned to CAL for a response rate of 60%. 
Of these, 9 were incomplete and 2 were late. Table 2.5 gives the 
demographic statistics of the 229 bilingual education teachers 
whose responses could be tallied. 
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Table 2.5 

Demographic Statistics of the Respondents 
to the Job-Re 1 at edness Survey (Bilingual Education) 



&j — Current Level of Assignment 
Level of Assignment Frequency 



Cumulative Cumulative 
Percent Frequency Percent 



Elementary 211 

Jun High/Mid School 3 

High School 3 

Other 2 
(Frequency Hissing ■= 10) 



96.3 
1.4 
1.4 
0.9 



211 
214 
217 
219 



96.3 
97.7 
99.1 
100.0 



& — Certificate or end orsement in bilingual education held? 

Cumulative Cumulative 
Certificate Frequency Percent Frequency Percent 



Yes 172 
No 46 
Invalid Resp 1 
(Frequency Missing ■ 10) 


78.5 
21.0 
0.5 


172 
218 
219 


78.5 
99.5 
100.0 


C. Years of Experience 
Experience Frequency 


Percent 


Cumulative 
Frequency 


Cumulative 
Percent 


1-2 years 48 
3-5 years 41 
6-10 years 60 
11-15 years 44 
16-19 years 16 
20 or more 7 
(Frequency Missing ■ 13) 


22.2 
19.0 
27.8 
20.4 
7.4 
3.2 


48 
89 
149 
193 
209 
216 


22.2 
41.2 
69.0 
89.4 
96.8 
100.0 


D. Level of Class Taught 









Class Level 



Frequency 



Cumulative Cumulative 
Percent Frequency Percent 



Early Childhood 39 
Grades 1-3 137 
Grades 4-6 35 
Invalid Resp 1 
(Frequency Missing * 17) 



18.4 
64.6 
16.5 
0.5 



39 
176 
211 
212 



18.4 
83.0 
99.5 
100.0 
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E, Highest Degree Held 

Cumulative Cumulative 
Highest Degree Frequency Percent Frequency Percent 





10 0 22 


10-0 


Bachelor's 140 


63.9 162 


74.0 


Master's 57 


26.0 219 


100.0 


(Frequency Hissing =10) 






F. Ethnicity 








Cumulative 


Cumulative 


Ethnic Group Frequency 


Percent Frequency 


Percent 


Hispanic 190 


87.2 190 


87.2 


Black 3 


1.4 193 


88.5 


White 24 


11.0 217 


99.5 


Other 1 


0.5 218 


100.0 



(Frequency Missing « 11) 



Cumulative Cumulative 
Sex Frequency Percent Frequency Percent 



Male 23 10.5 23 10.5 

Female 196 89.5 219 100.0 

(Frequency Missing - 10) 



Table 2.5 indicates that the typical respondent was a 
Hispanic female with a bachelor's degree and 6-10 years of 
teaching experience. She holds a certificate or endorsement in 
bilingual education and teaches in an elementary school (grades 
1-3). it may be interesting to note that only 12.8% of the 
respondents were not Hispanic. 

Table 2.6 presents the results of the job-relatedness survey 
for the bilingual education teachers. The tasks are ordered by 
average mean ranking. The standard deviation is presented in the 
second column as an indication of the agreement or disagreement 
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of the group on the mean ranking. The third column presents the 
approximate ACTFL level (l*=lntermediate, A=Advanced, S«Superior) 
of the speaking task. The final column shows the speaking task. 
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Table 2,6 

Results of the Job-Relatedness Survey 
for Bilingual Education 

Std. ACTFL 
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Table 


2.6 reveals that for the bilingual education group, 
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all of the speaking tasks were validated (i.e., the mean rating 
was above 3.50. Although the speaking tasks are generally ranked 
according to the ACTFL scale, it appears that many teachers rated 
items more on the basis of whether the speaking task is used in 
the classroom or not. Thus, the first task in rank order is 
"Giving Instructions," an Advanced level task which is clearly 
more difficult to accomplish than "Introduce Yourself," the 
second ranked task. Also, "Order a Meal" and "Make Purchases," 
fairly easy tasks, are rated towards the bottom and below "State 
Personal Point of View on a Controversial Subject," which is 
clearly more difficult. This intermingling of interpretations of 
the respondents task here, however, causes no serious problem 
since all the tasks were validated. 

The results of the bilingual education teachers indicated 
that TOPT items could be based on any of the speaking tasks in 
the survey. However, there were several items that the Spanish 
teachers did not validate. Since the Spanish TOPT would be used 
for both groups of teachers, and since from the survey it 
appeared that the final passing score for bilingual education 
teachers may be higher than that for Spanish language teachers 
(perhaps even at a Superior level), the final TOPT Spanish forms 
include one speaking task that was NOT validated by the Spanish 
language teachers. This speaking task is "Give a Professional 
Talk." It was included as a challenging task to bilingual 
education teachers because it could be answered very 
appropriately as a monologue, fitting nicely into the format of 
the TOPT. 



3. Development of the Trial Form of the TOFT 
The TOPT vas developed in two languages (French and 
Spanish) with four forms per language. This chapter describes 
how the first eight TOPT forms (Trial forms) were developed. 

3.1 Introducing the Project to the Test Advisory Committees 
On April 4, 1990, in Austin, Texas, a joint meeting was 
held of all members of both the French and Spanish Test Advisory 
Committees (TACs) . The membership of these committees was 
determined by the TEA and was intended to reflect interests of 
both teacher trainers and classroom teachers. In addition, the 
Spanish TAC membership reflected the concerns of both Spanish 
language teachers and bilingual education teachers. 

At this initial meeting, Dr. Charles w. Stansfield, Project 
Director at CAL, introduced the TAC members to the project. He 
also led discussion of the speaking tasks to be included in the 
job-relatedness survey (Chapter Two) . Below are the names and 
affiliations of the members of the two TACs. 



30 

36 



French Test Advisory Committee Members 



Dr. Tovnsend H. Bowling 

Ms. Cathy Champagne 

Ms. Betty Clough 

Dr. Maurice G.A. Elton 

Dr. Arthur Gionet 

Mr. David Long 

Dr. Joan H. Manley 

Ms. Fran Maples 



University of Texas, San Antonio 
Cypress-Fairbanks ISD 
Austin ISD 
Southern Methodist University 
University of North Texas 
Spring Branch ISD 
University of Texas, El Paso 

Richardson ISD 



Spanish Test Advisory Comm ittee Members 



Dr. 
Ms. 
Dr. 
Dr. 
Dr. 
Ms. 
Ms. 
Ms. 
Ms • 
Ms. 
Dr. 



Luz Elena Nieto 
Annette Ortega 
Maggie Stovall 
Marion R. Webb 



George M. Blanco 
Mary Diehl 



Ellen de Kanter 
George Gonzalez 



Barbara Gonzalez Pino 
Claudina Hernandez 
Carmen Muftoz 



University of Texas, Austin 

Round Rock ISD 
University of St. Thomas 
University of Texas, Pan American 
University of Texas, San Antonio 

Alice ISD 
Pharr-San Juan-Alamo ISD 

El Paso ISD 
Amarillo ISD 
Alamo Heights ISD 
Houston Baptist University 



3.2 Developm ent of the Initial Test Items 

An item-writing team composed of staff at CAL experienced 

in writing items for SOPIs (known as the Local Test Development 

Team — LTDT) worked together to develop items for the initial four 

forms for each language of the TOPT. The team members were: 

Dr. Charles W. Stansfield Project Director 

Mr. Dorry Mann Kenyon Project Coordinator 

Dr. Mary Lee Scott Item Writer 

Mr. John Karl Item Writer 

Mr. Daniel Kennedy Item Writer 

Ms. Ruth Ephraim Project Artist 

The LTDT made every effort to keep in mind the examinees 

who would be taking the TOPT in order to construct items 

accessible to them. The LTDT assumed the typical TOPT examinee 
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would: 1) have an interest in teaching, 2) be familiar with 
school and college life, 3) have some interest in language, and 
4) have some familiarity with the state of Texas. Items of 
either a personal nature or requiring some factual knowledge took 
into account the above assumptions. The LTDT made every effort 
to avoid items that were too personal (and thus be uncomfortable 
for some examinees to answer) cr too specific (and thus be beyond 
the grasp of some examinees) . 

The LTDT worked intensively between the beginning of April 
and the beginning of June, 1990, to develop the items for the 
four forms, in the beginning, items were written following the 
specifications used to develop earlier CAL sOPIs. Items were 
written so that they could be used on both French and Spanish 
forms with appropriate modifications. When the results of the 
survey (Chapter Two) were made available in May, it became clear 
that more difficult items could be used on the Spanish TOPT than 
on the French TOPT, since the bilingual education teachers had 
validated all of the Superior level speaking tasks and the 
Spanish teachers most of them, while the French teachers had 
validated only two Superior level speaking tasks, i fter this 
date, then, the LTDT wrote items for a larger number of Superior 
level speaking tasks for the Spanish TOPT than had been found on 
CAL's earlier SOPIs. 

After each item was written, it was reviewed, revised and 
rewritten until the Project Director and Project Coordinator were 
satisfied with its quality. Once all items were completed, they 
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were carefully selected for placement into four forms that would 
be parallel in terms of speaking tasks covered, number of 
education/non-education related items, difficulty of item 
prompts, and variety of topics covered. Special care was taken 
that a variety of topic areas were covered on each form and that 
no form contained more than one item in any topic area (e.g. 
computers) . Since the TOPT is an assessment of general speaking 
ability, the context of the items on the TOPT could not be 
restricted to only school-related settings and language usage. 
However, in light of the population of bilingual educators who 
would be taking the test and for whom Spanish language usage in 
the context of the classroom is primary, effort was made to 
ensure chat approximately 50% of the items on the Spanish TOPT 
were directly school or education related. The contexts of items 
on the French TOPT were more varied, not being under a similar 
constraint. 

By the end of May, four forms of the TOPT in each language 
were assembled. At this point, the TEA reviewed the compiled 
TOPT forms. Their suggested revisions were then incorporated 
into the tests. 
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3.3 Review of Test Forma by the Bias Review Committees 

The TEA nominated a Bias Review Committee (BRC) for each 
test. The BRCs were composed of the following Texas teachers: 



Bias Review Committ ee - Spanish 



Ms. Connie Kunkel 
Ms. Sylvia Wade 



Cypress -Fairbanks ISD 
Northside ISD 



Bias Revie w Committee - French 



Ms. Jacqueline Hullaby 
Mr. Adalberto Saenz 



Houston ISD 
Alice ISD 



On June 4, the two BRCs convened in Austin and carefully 
reviewed the TOPT forms, checking for any potentially bias- 
related problems in the items. Their suggestions for r^ /is ions 
were noted and brought to the attention of the TACs. 

The Spanish TAC met on June 5 and 6 in Austin; the French 
TAC meeting followed on June 7 and 8. At these meetings, the TAC 
members were first presented with the results of the job- 
relatedness survey. They then proceeded to collectively review 
the forms item by item, commenting on each item's 
appropriateness, accessibility to all candidates, clarity, and 
potential for eliciting responses displaying use of the targeted 
speaking tasks. Parallel items across the forms were reviewed 
together so that TAC members could comment on their comparability 
in terms of their wording and difficulty. The TAC committee 
members also considered all the recommendations and comments of 
the BRCs. 
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3.4 Revision of Trial Forms bv the Local Test De velopment 
Committee 

Immediately following the BRC and TAC meetings, the LTDT 
revised the eight TOPT trial forms according to all revisions 
corporately accepted during the meetings. Following this, the 
eight revised forms of the test were sent to the TEA for 
approval . 

3.5 Preparation of the TOPT fo r Trial ina 

Once the TEA had approved the Trial forms of the TOPT, the 
tapescript (containing the test directions, items, and native 
language prompts) were recorded at a professional recording 
studio. The test booklets were prepared at CAL, together with 
the forms that would be used to collect data during the trial ing. 
These are described in Chapter Four. 
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4. Trial inc* the TOPT 
This chapter describes how the TOPT was trialed on 
examinees from throughout the state of Texas. It describes the 
procedures used to recruit examinees, administer the test, 
collect data, and analyze the results of the trial ing. 

4.1 The Purpose of Trial ing 

For a performance test such as the TOPT, a careful 
examination of its ability to elicit a ratable speech sample is 
needed. Trial ing is the method used to study the TOPT's ability 
to elicit ratable speech. Trial ing may be described as an 
intensive "qualitative" approach to test development (as opposed 
to an extensive "quantitative" approach based on the piloting of 
the test and calculation of item st • tics). Trialing produces 
feedback from examinees, observers aim raters to study important 
characteristics of a performance-based test, such as the ability 
of each item to allow examinees to demonstrate their skill, the 
adequacy of the time allotted for the performances (in the case 
of the TOPT, the length of the pauses between items) , the clarity 
of the instructions for each item, the perceived appropriateness 
and "fairness" of each item, the interpretability of drawings or 
pictures used, and the usefulness of the performance (for the 
TOPT, the speech elicited) in the determination of a rating. 
Feedback from examinees, observers and raters further helps 
ensure that the forms are comparable in difficulty. 



4.2 Recruitin g Examinees for the Trialing 

From the outset of the projt , both the TEA and CAL 
recognized that it might be difficult to recruit the numbers and 
types of examinees desired for the trial ing during the summer 
when very few college courses were being offered. We were aiming 
for a group of trialing examinees whose composition would reflect 
those who would be taking the actual test once the TOPT became a 
requirement. This latter group would be predominantly composed 
of individuals from throughout Texas currently enrolled in a 
teacher preparation program who were preparing to teach French, 
Spanish, or bilingual education. 

In light of the difficulties of finding large intact 
populations of such examinees during the summer, particularly for 
French, the TEA and CAL set the following goals. Each of the 
four forms of the French test would be administered to 8 
individuals, each taking one form of the test. In total, there 
would be 32 French examinees, half of whom would be pre-service 
teachers (i.e., individuals preparing to teach but not yet 
certified) . Each form of the Spanish test would be administered 
to 20 individuals: 10 preparing to teach Spanish and 10 bilingual 
education. Each examinee would take one form of the test 
resulting in a total of 80 Spanish examinees, half of whom would 
be pre-service teachers. 

The TEA and CAL adopted various methods to recruit 
examinees. First, with the input of the members of the two TACs, 
several trialing sites throughout Texas were chosen. The sites 
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ultimately used for trial ing were: 



El Paso 

Austin 

Arlington 

Hurst 

Edinburg 

San Antonio 

Houston 



University of Texas at El Paso 
University of Texas at Austin 
University of Texas at Arlington 
Tarrant County Community College 
Pan American University 
University of Texas at San Antonio 
University of St. Thomas 



Then, again with the help and input of TAC members, CAL 
generated lists of potential examinees. These were: 



CAL sent these individuals information about the trialing 
and a return postcard on which they were to indicate their 
willingness to participate. An example of this invitation to 
participate is found in Appendix B. 

In some cases (particularly in bilingual education), there 
were intact groups that were invited to participate. For Spanish 
and bilingual education, these were university classes in session 
during the summer, whose professors were contacted by CAL. For 
French, there was a summer institute for French teachers being 
held in Arlington whose leader was contacted. The professors of 
these groups announced the trialing and encouraged students to 
participate. They then sent CAL a list of the names of students 
in their classes who were willing to participate in the trialing. 

4 . 3 Participation in the Trialing 

In terms of examinee numbers, results exceeded the original 
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individuals known personally by TAC members, or 
individuals identified by teaching faculty at schools 
in the testing site areas, or 

individuals identified by certification offices at 
schools in the testing site areas. 
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goals. 119 examinees took the Spanish TOPT, while 41 examinees 
took the French TOPT. Table 4.1 below gives the numbers of 
examinees at each of the seven trialing sites. 



Table 4.1 

Numbers of Examinees at Each Trialing Site 









Cumulative 


Cumulative 


CITY 


Frequency 


Percent 


Frequency 


Percent 


El Paso 


11 


6.9 


11 


6.9 


Austin 


28 


17.5 


39 


24.4 


Arlgton 


17 


10.6 


56 


35.0 


Hurst 


24 


15.0 


80 


50.0 


Edinburg 


56 


35.0 


136 


85.0 


San Ant 


17 


10.6 


153 


95.6 


Houston 


7 


4.4 


160 


100.0 



Table 4.1 .indicates that there was not an even distribution 
of examinees at all s*tes. There were relatively few in Fi Paso 
and Houston, while there was a considerably larger number at 
Edinburg, due in large measure to the keen interest in the 
project of bilingual education professors at both Pan American 
University and University of Texas at Brownsville. 

Table 4.2 gives the background information on the examinees 
who took the French TOPT. 



Table 4.2 

TOPT French Examinees: Descriptive Data 



A. TOPT Form Taken 



Cumulative Cumulative 

FORM Frequency Percent Frequency Percent 

Form A 10 24.4 10 24.4 

Form B 10 24.4 20 48.8 

Form C 10 24.4 30 73.2 

Form D 11 26.8 41 100.0 
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Trial ina Site 

Cumulative Cumulative 
CITY Frequency Percent Frequency Percent 





Austin 


3 


7.3 


3 


7.3 




Arlgton 


17 


41.5 




48.8 




Hurst 


15 


36.6 


35 


85.4 




Edinburg 


3 


7.3 


38 


92.7 




San Ant 


2 


4.9 


40 


97.6 




Houston 


1 


2.4 


41 


100.0 




Current Status in Respect to Teach ina 














Cumulative 


Cumulative 




STATUS Frequency 


Percent 


Pre cm e ncv 


Percent 




Pre-Serv 


6 


15.0 


6 


15.0 




In-Serv 


15 


37.5 


21 


52.5 




Other 


19 


47.5 


40 


100.0 




(Frequency Missing «= 


1) 






Df 


Area of Certification 
















Cumulative 


Cumulative 




CERT 


Frequency 


Percent 


Frequency 


Percent 




El Fren 


1 


2.5 


1 


2.5 




Sec Fren 


16 


40.0 


17 


42.5 




Sec Span 


2 


5.0 


19 


47.5 




Fr & Sp 


7 


17 . 5 


26 


65.0 




Other 


14 


35.0 


40 


100.0 




(Frequency Missing = 


1) 






E. 


Ethnicity 


















Cumulative 


Cumulative 




ETHNIC 


Frequency 


Percent 


Frecruencv 


x JL ww J 1 w 




Black 


1 


2.4 


1 


2.4 




Hi span 


3 


7.3 


4 


9.8 




Other 


5 


12.2 


9 


22.0 




White 


32 


78.0 


41 


100.0 


F- 


Sex 


















Cumulative 


Cumulative 




SEX 


Frequency 


Percent 


Frequency 


Percent 




Female 


34 


82.9 


34 


82.9 




Male 


7 


17.1 


41 


100.0 



40 
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G. Self Rating of Ability on the ACTFL Scale 



Cumulative Cumulative 
Frequency Percent 



SELFRATE 



Frequency 



Percent 




5.0 
15.0 
25.0 
12.5 
35.0 

7.5 



2 
8 
18 
23 
37 
40 



5.0 
20.0 
45.0 
57.5 
92.5 
100.0 



(Frequency Missing ■ 1) 



Table 4.2 reveals that an almost equal number of examinees 
took each form of the French TOPT. 41.5% of the total examinees 



teachers at Arlington. Only 15% of the total number of French 
examinees were pre-service teachers. Despite this, there was a 
wide range of abilities in the sample. On the self -rating (see 
Section 4.5 for more information about this), 42.5% were in the 
Intermediate level, 37.5% at the Advanced level, and 20% Superior 
or above. It may be noted that on the self-rating, 42.5% rated 
themselves under the passing score of Advanced, while 57.5% rated 
themselves at Advanced or above. 21.9% of the examinees were 
members of minority groups and 17.1% were male. French TAC 
members felt that these last figures were representative of the 
population of in-service French teachers in Texas. 

Table 4.3 presents the background information on the 
examinees who took the Spanish TOPT. 



were part of a special summer program for in-service French 
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Table 4.3 

TOPT Spanish Examinees: Descriptive Data 



A- 


TOPT Form 


















Cumulative 


Cumulative 




FORM 


Frequency 


Percent 


Frequency 


Percent 




Form A 


26 


21.8 


26 


21.8 




Form B 


28 


23.5 


54 


45.4 




Form C 


38 


31.9 


92 


77.3 




For© D 


27 


22.7 


119 


100.0 


Pi 


Trial ina Site 


















Cumu 1 at ive 


Cumulative 

m^ !■ IM mm mm mm %» mm W 9 mjm 




CITY 


Frequency 


Percent 


Frequency 


Percent 




El Paso 


11 


9.2 


11 


9.2 




Austin 


25 


21.0 


36 


30.3 




Hurst 


9 


7.6 


45 


37.8 




Edinburg 


53 


44.5 


98 


82.4 




San Ant 


15 


12.6 


113 


95.0 




Houston 


6 


5.0 


119 


100.0 


C 


Current Status in Respect to Teachina 














Cumulative 


Cumulative 




STATUS 


Frequency 


Percent 


Frequency 


Percent 




Pre-Serv 


72 


63.2 


72 


63.2 




In-Serv 


19 


16.7 


91 


79.8 




Other 


23 


20.2 


114 


100.0 




(Frequency Missing = 


5) 






P. 


Certification 


















Cumulative 


Cumulative 




CERT 


Frequency 


Percent 


Frequency 


Percent 




El Span 


11 


9.6 


11 


9.6 




Sec Span 


26 


22.6 


37 


32.2 




Bil Ed 


66 


57.4 


103 


89.6 




Other 


12 


10.4 


11J 


100.0 



(Frequency Missing 
E. Ethnicity 



4) 



ETHNIC Frequency Percent 



Cumulative Cumulative 
Frequency Percent 



Black 

Hispan 

White 



1 

95 
23 



0.8 
79.8 
19.3 



1 
96 
119 



0.8 
80.7 
100.0 
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F. Sex 

Cumulative Cumulative 
SEX Frequency Percent Frequency Percent 



Female 
Male 



88 

31 



73.9 
26.1 



88 

119 



73.9 
100.0 



G. Self Rating on the ACTFL Scale 



SELFRATE 



Frequency 



Percent 



Cumulative Cumulative 
Frequency Percent 



High-Sup 13 

Sup 20 

Adv+ 40 

Adv 18 

Int-H 15 

Int-L/M 6 

Novice 1 



11.5 
17.7 
35.4 
15.9 
13.3 
5.3 
0.9 



13 
33 
73 
91 
106 
112 
113 



11.5 
29.2 
64.6 
80.5 
93.8 
99.1 
100.0 



(Frequency Missing - 6) 



Table 4.3 reveals that although an unequal number of 
examinees took each form of the Spanish TOPT, no form was taken 
by less than 26 examinees, which is 30% greater than the initial 
goal. 44.5% of the total number of examinees took the TOPT at 
Edinburg, which, as mentioned above, was due to the interest of 
local professors. Unlike for the French TOPT, over half (63.2%) 
were pre-service teachers, while only 16.7 were in-service. On 
the basis of the self-ratings, there was less of a distribution 
of ability than on the French TOPT, with over half (51.3%) at one 
level (Advanced), with 0.9% at the Novice level, 18.6% at the 
Intermediate level, and 29.2% at the Superior level. 80.5% rated 
themselves above the passing score of Advanced, with only 19.5% 
under the passing score. Slightly more than half (57.4%) were 
involved in bilingual education. Hispanics made up approximately 
79.8% of the total sample, and males were 26.1%. It may be noted 
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that of the Spanish TOPT examinees indicating bilingual education 
as their field of certification, 87.9% were Hispanic and 19-7% 
were male; of those not indicating bilingual education, 69.8% 
were Hispanic and 34% were male. 

Appendix C contains a breakdown of the descriptive data of 
the TOPT trialing examinees by language and form. 

4.4 Administering the TOPT 

The TOPT was trialed during the period from June 26 to July 
3, 1990. At each trialing site, for the convenience of the 
examinees, the TOPT was administered four times over a period of 
a day. The only exceptions to this were at Arlington, where it 
was administered only once to the group of in-service French 
teachers taking part in a summer institute, and Hurst, where it 
was administered only twice during a half -day stssion. 

Both the test times and the presence or absence of 
simultaneous administrations of the TOPT depended on the 
equipment of the language laboratory at each testing site and the 
number of examinees expected at that site for each test. CAL 
made every effort to make the test administration as convenient 
as possible for all parties involved. Testing periods were 
scheduled in one and one-half hour blocks. 

During the first part of the testing session, lasting 
approximately 50 minutes, the examinees took the TOPT. During 
the second part, the examinees recorded their feedback on the 
test on two data collection forms. These are described in 
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section 4.5. Examinees spent between 30 and 90 minutes 
completing these two data collection forms. 

4.5 Data Collection Purina Trialina 

CAL collected feedback on the test from two main and one 
secondary sources. The first main source of feedback was from 
the examinee. After the test was administered, each examinee 
completed a two-part evaluation form eliciting quantitative 
ratings and qualitative written comments on both specific TOPT 
items and on the test in general. 

The first part of the form, which appears in Appendix D, 
collected quantitative data. On a machine-readable answer sheet 
developed at CAL, examinees first gave background and demographic 
information and assigned themselves a rating on a simplified 
ACTFL scale. These self-ratings served as a "ballpark" estimate 
of the examinee's TOPT score and were used later to determine 
which Spanish tapes would be listened to completely by raters 
(see below) . The results of the background, demographic, and 
self-rating responses were presented in section 4.3 above. The 
machine-readable response sheet also included two statements for 
every item (the opening conversation being counted as one item) : 
the first dealt with the adequacy of the time allowed for the 
item, the second with its overall quality. In addition, for the 
five picture items, one statement per item dealt with the 
perceived clarity of the picture. Three statements about the 
nature of the test in general completed this form. To each of 
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these 40 statements, examinees had to indicate, on a scale of 1 
to 5 (with 5 being highest), their degree of agreement. 

The second part of the examinee response form, which 
appears in Appendix E, collected qualitative data. For any 
statement to which the examinee gave a lower rating on the first 
part of the form, this part requested the examinee to explain why 
by writing his or her comments in spaces provided. At the end of 
the form examinees were requested to write about any concerns 
they had about the test that were not addressed elsewhere. 

The second main source of trial ing feedback came from 
"judges." These individuals, familiar with the ACTFL scale, 
listened to examinee tapes, foe; ring on the quality of the speech 
elicited by each item on the test for making a rating for that 
examinee. They recorded their comments on a special form, a copy 
of which is found in Appendix F. On this form, they marked on a 
scale of 1 to 3 the usefulness of the speech sample elicited by 
each item in determining that examinee's proficiency level. For 
each item and each examinee, they also indicated the 
appropriateness of the time allowed for the response. Moreover, 
the judges also noted any potential problems with specific items 
based on their analysis of the examinee's performance. Each tape 
was listened to by one judge. Two judges listened to all of the 
French tapes, while 80 (20 for each form, half bilingual and half 
Spanish) of the 119 Spanish tapes were listened to by one of six 
judges. Of the Spanish tapes, all tapes made by non-Hispanics 
and all tapes made by examinees who gave themselves a rating 
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lover than Advanced were included in the 80 tapes. To complete 
the number of Spanish examinees to be listened to, tapes were 
selected at random from those remaining. 

A final (albeit minor) source of feedback during trial ing 
came from observers. When possible, TEA foreign language staff 
present during the trial ing administration served as observers. 
The observers concentrated on listening to the response of one 
individual and wrote comments on a special form, a copy of which 
is found in Appendix F (the same form used by the judges) . 
Because few individual examinees were tracked by observers, 
observer comments did not account for much feedback data. 

4.6 Results Qf the Trailing 

The feedback collected during the trial ing was used to 
guide the post-trialing test revision process. Originally, the 
procedures called for using the quantitative data collected (the 
ratings from the examinees and judges) to pinpoint problem areas 
and then use the examinees* written comments (qualitative data) 
to inform the specific revisions that were to be made. In 
actuality, quantitative data was reviewed and all written 
comments for all items were read before determining whether any 
revision should be made. 

4.6.1 Results of the Examinee Data Form. Part 1 

A data file was created at CAL from the machine-readable 
examinee response forms. The forms were read with a NCS Sentry 
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3000 scanner and the SCANTOOLS software program was used to 
create a data file that was imported to and then analyzed with 
SA5. Each of the eight test forms was analyzed separately. 
Since 5 was the highest rating, any statement with an average 
score of 3.50 or below identified a problematic TOPT item. Those 
with a score above 3.50 but below 3.75 identified TOPT items that 
were carefully looked at to determine if any potential problems 
existed. Statements with a mean rating above 3.75 were 
considered to identify TOPT items not needing serious revision. 
The mean rating for each statement for each TOPT form is 
presented in Appendix G. Items marked by an * in Appendix G are 
those above 3.50 but below 3.75. Items marked by ** are those at 
3.50 and below. Table 4.4 contains the number of items that were 
rated in each category for each form. 



Table 4.4 

Number of Items in Each Rating Category 

Per TOPT Form 



SPANISH FORM A (n=26) 

Above 3.75 
3.51-3.75 
Below 3.50 

SPANISH FORM B (n=28) 

Above 3.75 
3.51-3.75 
Below 3.50 

SPANISH FORM C (n-38) 

Above 3.75 
3.51-3.75 
Below 3.50 



General 

8 
8 
0 

General 

5 
7 
4 

General 

2 
1 

13 



Time 

2 
8 
6 

Time 

3 
3 

10 

Time 

2 
4 
10 



Pictures 

4 

1 
0 

Pictures 

3 
1 
1 

Pictures 

3 
1 
1 
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SPANISH FORM D (n=27) 



General 



Tine 



Pictures 



Above 3.75 
3.51-3.75 
Below 3.50 



10 
5 
1 



9 
6 
1 



4 
1 

0 



FR ENCH TOm ft (n*10) 

Above 3.75 
3.51-3.75 
Below 3.50 

FRENCH FORM B (n*10) 

Above 3.75 
3.51-3.75 
Below 3.50 

FRENCH FORM C (n=10) 

Above 3.75 
3.51-3.75 
Below 3.50 

F RENCH FORM D (n=ll) 

Above 3.75 
3.51-3.75 
Below 3.50 



General 

14 
2 
0 

General 

6 
2 
8 

General 

14 
1 
1 

General 

11 
2 
3 



Tine 

15 
1 
0 

Tine 

10 
3 
3 

Tine 

11 
4 

1 

Tine 

5 
6 
5 



Pictures 

4 

1 
0 

Pictures 

3 
1 
1 

Pictures 

4 
1 
0 

Pictures 

4 

1 
0 



The above table indicates that each form had its own pattern 
of responses, with the Spanish TOPT, for three of the four 
forms, there were no or few nean ratings below 3.50 to the 
general statenent: none for Form A, 1 for Form D and 4 for Form 
B. For Form C, however, 13 statements had nean ratings below 
3.50. In general, quite a few of the time statements received 
low ratings (examinees generally wanted more time) : 6 for Form A 
and 10 for both Forms B and C. However, only 1 time statement 
for Form D received a rating below 3.50. with the French TOPT, 
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ratings were generally higher than with the Spanish test. For 
Form A, no statement received a rating below 3.50, while for Form 
C, only 2 did. On the other hand, Form D had 8 statements below 
3.50 while Form B had 12. For the French TOPT, time did not seem 
much of a problem on Forms A, B, and C, though for Form D only 5 
time statements received a rating above 3.75. 

These numbers indicate that revisions needed to proceed item 
by item. Because a different group of examinees took each form, 
it would be unwise use this data to make general inferences about 
the quality of each trialing test form. 

4-6.2 Results of the Examinee Data Form. Part 2 

All of the written comments were coded as to the TOPT form 
and item they referred to and as to the degree of negativity 
expressed in them. Then they were typed into a word-processing 
database. A printout of comments pertaining to each item was 
then produced. An example of one such printout appears in 
Appendix H. 

The written comments primarily served to inform the 
specific revisions that needed to be made. In analyzing the 
printout of comments, we considered the various classifications 
into which comments generally fell. First were positive 
comments. These were substantial, even though examinees were 
requested to comment only if there was a problem with the item. 
Second were concurring comments that pointed out some flaw 
requiring revision, usually corroborating lower mean ratings of 
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statements. Third were unique negative comments suggesting 
helpful revisions, even when not corroborated by the comments of 
others or by the ratings. Fourth were unique negative comments 
with off -target suggestions, indicating that the examinee's 
problem with the item was not due to any attribute of the item 
per se. An example of one such comments was "Current events is a 
topic that should be left out. All other situations were 
appropriate but for a full time student, current events may be an 
unknown area." By far the most common suggestion was to increase 
the amount of time allowed to prepare and give the response. In 
general, the examinees' written comments were quite helpful in 
making revisions. 

4.6.3 Results of the Judge's Response Sheet f Quantitative! 

The judge's response sheet contained two types of 
quantitative data: data on the quality of the speech sample 
elicited by the item and data on any perceived time problems with 
the item. 

Appendix I contains the quality ratings by TOPT form and 
item. Since the maximum possible rating was 3, these mean 
ratings were generally quite high. Table 4.5 gives the average 
item quality rating for each form. 
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Table 4.5 

Average Item Quality Rating for Each Form 



Form 


Average Rating 


Range 


French A 


2.69 


2.38 - 3.00 


French B 


2.24 


1.88 - 2.50 


French C 


2.93 


2.75 - 3.00 


French D 


2.87 


2.67 - 3.00 


Spanish A 


2.73 


2.57 - 2.80 


Spanish B 


2.59 


2.35 - 2.78 


Spanish C 


2.57 


2.38 - 2.67 


Spanish D 


2.71 


2.55 - 2.85 



Table 4.5 indicates that across items, the mean item quality 
is quite high, particularly when one remembers that 1 meant the 
quality was poor, 2 meant average and 3 meant excellent. 
Generally, none of the items were perceived to be particularly 
problematic by any of the judges. The average rating of French 
Form B seems unusually low; however, that is explained by the 
fact that the majority of the Form B tapes were judged by one of 
the two French judges who tended to award everything "2" unless 
it was exceptional. In other words, this judge was more severe 
in her quality ratings. 

Appendix J contains an example of the judges* time data by 
TOPT form and item. In making revisions, convergence in 
additional time needed (or too much time) across raters and 
examinees was sought. Judges were instructed to only mark "too 
little time" if the time allotted did not allow the examinee to 
demonstrate whether he or she could handle the targeted speaking 
task. In other words, they were not to mark "too little time" if 
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the examinee did not finish his or her answer but gave a clear 
indication of ability (or lack thereof) to handle the speaking 
task. In the vast majority of cases, obvious patterns developed, 
as illustrated by the example Appendix J. Response time was 
almost always lengthened (it was shortened in a few cases), so 
that 90% of the examinees would have enough (projected) time to 
complete the item. 

4.6.4 Results of the Judge's Re sponse Sheets f Qualitative) 

The comments of the judges were mostly notes on the 
idiosyncracies of examinees. In general such comments were not 
particularly helpful in revising items, though in some cases the 
judges 1 comments did shed light on a problem area or suggested a 
solution. An example of judge's comments for one item can be 
found in Appendix K. 
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5. Development of wie Final Form 
This chapter describes how the final forms of the TOPT was 
developed, based on the results of the trialing. 

5.1 Revisions of the Trial Forms 

After CAL collected and analyzed all data from the trialing, 
the members of CAL's LTDC reviewed the results item by item and 
made revisions as appropriate. These were then sent to the TEA 
for their inspection and comments. 

Between August 18 and August 21, 1990, final meetings of the 
BRCs and TACs took place. At these meetings, a summary of the 
trialing results was presented. The BRCs reviewed the revised 
items for indication of bias and made comments and suggestions 
for revisions. The TACs then discussed all revised items and 
acted on comments made by the BRCs. Final wording and final 
revisions were the basis ol group decisions made at these 
meetings, the outcome of which became the final version of each 
TOPT form. 

Following these meetings, CAL and the TEA decided that 
Trialing Form B would become the disclosed form of the TOPT. 
Examples for the TOPT Registration Manual would be taken from 
this form, and this form would appear in its entirety as the 
Practice Test of the TOPT Test Preparation Kit . CAL also decided 
that the three remaining forms of the TOPT would be renumbered. 

During the fall and winter of 1990, the TOPT Master Tapes 
created for the trialing were edited to reflect the final 
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collectively agreed upon revisions. These were then carefully 
reviewed by the TEA, reedited if necessary, and then accepted as 
final. The project artist also made the necessary changes to the 
pictures. Once these were approved by the TEA, the pictures were 
drawn in pen and ink. The pen and ink drawings were once again 
reviewed by the TEA. After a few minor changes were made, the 
pictures were readied for publication, along with the TOPT test 
booklets. 
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6. Content validation 
This chapter describes how the content of the final forms of 
the TOPT was validated. 

6.1 Content Validation 

At the foundation of the TOPT's content lie the speaking 
tasks, based on the ACTFL guidelines, which were validated during 
the job-relatedness survey (see Chapter Two) . Each item on the 
TOPT was written to elicit language in response to one of these 
tasks. In order to investigate whether the items on the final 
forms of the TOPT did indeed match the speaking tasks which 
served as the specifications for the items, three separate 
content validation studies were undertaken in the fall of 1990: 
one for French, one for Spanish, and one for bilingual education. 
The French and Spanish studies were held jointly on Friday, 
October 19, during the Fall 1990 Texas Foreign Language 
Association Convention in Fort Worth. The bilingual education 
study was held during the Fall 1990 Texas Association of 
Bilingual Educators meeting, on Thursday, November 1, in Lubbock, 
Texas. 

For each group, the TEA submitted to CAL a list of qualified 
judges who were representative of both teachers and teacher 
trainers. CAL prepared letters of invitation to each individual, 
inviting him or her to participate in the study. The following 
individuals served as judges in the content validation 
studies: 
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French 



Dr. Phyllis Nimmons 

Mr. David Hardy 

Ms. Margie Rodger s 

Dr. Marie Christine Koop 



Houston Baptist University 
Carroll ton-Fanners Branch ISD 

Ector County ISD 
University of North Texas 



Spanish 



Dr. Armando Armengol 

Ms. Teresa Garcia 

Ms. Rose Potter 

Mr. Steve Black 

Dr. Judith Marquez 



University of Texas, El Paso 

Mission ISD 
Eanes ISD 
Angel ton ISD 
University of Houston, clear Lake 



Bilingual Education 



Ms. Rosa M. Chahin 
Ms. Virginia Moore 
Ms. Elizabeth Martin 
Dr. Juan Lira 



Houston ISD 
Midland ISD 
Grand Prairie ISD 
Laredo State University 



The task of the judges was to examine each item on the three 
operational forms of the TOPT and determine whether it elicits 
the speaking task (e.g., support an opinion, give directions, 
state advantages and disadvantages, etc.) specified for it. The 
judges read the TOPT items in specially prepared notebooks 
containing each TOPT item on a separate page with the item's 
speaking task at the top of the page. They recorded their 
responses on a separate sheet. The instructions to the judges 
and a copy of -ne Judge's Response Sheet can be found in Appendix 
L. 

6.2 Results 

All of the TOPT items were validated by the content 
validation studies as matching the targeted speaking task for the 
item. The great majority of items presented no problem to the 
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judges. In almost all of the cases where judges gave a negative 
mark, their own comments revealed that their responses were off- 
target; i.e., the negative mark was awarded for a reason other 
than that the item did not match the speaking task. In only one 
case (Topic 1 for French) was a problem found with the speaking 
task description used (for one of the three forms). Labeled in 
the content validation study test item booklet as "Describe 
Personal Activities," this item was originally intended to cover 
the intermediate level speaking tasks of providing a description 
of persons, places, and things familiar to the individual (i.e., 
net only of personal activities, such as hobbies and pastimes) . 
The speaking task for this item has since been relabeled as "Give 
a Description of a Personal Mature." 



58 

64 



7. Standard Setting Study 
This chapter describes how passing score standards for the 
TOPT were set. In order to provide additional data to assist the 
TEA and the Texas State Board of Education in setting passing 
scores for the TOPT, three separate standard setting studies, 
following the model described in Livingston (1978) and adapted by 
Powers and Stansfield (1982), were carried out in the fall of 
1990, concurrently with the content validation studies described 
in Chapter Six. These studies required a sampling of examinee 
performances and a panel of judges to rate the performances as 
acceptable or unacceptable. 

7.1 Preparation of th e Standard Setting Master Tape 

Before the study could be conducted, it was necessary to 
prepare a master tape containing TOPT performances of examinees 
at different levels of speaking proficiency. 

7.1.1 Initial Selection of Representatives of Various Levels 
of Speaking Proficiency 
The judges who listened to the examinee response tapes 
following the trialing (see Section 4.x) gave a preliminary 
rating to each examinee they listened to. Additionally, each 
examinee provided a self-rating during the trialing. Originally 
CAL had planned to use these two pieces of information to choose 
tapes that, upon receiving complete rating, would provide three 
examples at each of seven ACTFL levels (IL, IM, IH, A, A+, S, 
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S+ a ) . This would have made a total of 21 examples. In the first 
step, CAL looked for five potential examples at each level, for a 
total of 35. However, there were virtually no intermediate Low 
level tapes in the Spanish set of tapes. Thus, the total number 
of tapes selected for the first step for Spanish was 31. All 32 
audible French tapes were included in this first phase of the 
study . 

7.1.2 Ratings bv Texas ACTFL-Certi f ied Raters 

The TEA and CAL jointly chose two prominent Texas ACTFL- 
certified raters for each language to score the selected trial ing 
tapes within a period of three weeks. Dr. George Blanco of the 
University of Texas at Austin and Dr. Vickie Contreras of the 
University of Texas, Pan American, served as the Spanish raters; 
Dr. Joan Manley of the University of Texas at El Paso and Ms. 
Mary Huggins of Round Rock ISD served as the French raters. 
Appendix M contains the instructions to these raters and as an 
example, a copy of the form used to record their scores for 
French and Spanish Form B. 



3 For practical purposes, ACTFL levels were assigned a 
numerical classification as follows: 



Novice 




0 


Intermediate-Low 


IL 


1 


I ntermed iate-Mid 


IM 


2 


Intermediate-High 


IH 


3 


Advanced 


A 


4 


Advanced+ 


A+ 


5 


Superior 


S 


6 


High-Superior 


S+ 


7 



The order of the tapes to be rated was prescribed. The 
raters listened to all tapes for each form, beginning with Form 
A. Within each form, examinees were sequenced by social security 
number. Raters gave each examinee a rating on each item and the 
opening conversation as a whole, and also rated each one 
holistically. 

Blanco and Contreras were found to have few holistic and 
item level score disagreements. Manley and Huggins, on the other 
hand, had few agreements but correlated very highly, with Huggins 
being consistently more lenient than Manley. At the lower 
levels, the difference between them was about one tep on the 
ACTFL scale; at the higher levels, the difference reached two or 
three steps at times. Thus, where Manley awarded an Intermediate 
Low, Huggins typically awarded an Intermediate Mid? where Manley 
awarded Intermediate Mid, Huggins typically awarded Intermediate 
High; and where Manley awarded an Advanced or Advanced Plus, 
Huggins typically awarded a Superior or High Superior. 
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7.1.3 constructi on of the First Master Tape 

In constructing the master tape for the standard setting 
study, the goal was to have each individual examinee represented 
by three segments (responses) which would be indicative of a 
certain ACTFL level. The segments chosen for each examinee 
proceeded as follows. Data from the ratings of the Texas ACTFL 
raters were entered into a database. Then the matches between 
the two item ratings for each examinee cn all items were 
identified; that is, items were identified where the two raters 
agreed on the item-level rating. In the case of French, matches 
were also identified where Huggins was consistently one level (or 
two) above Manley. 

Once these item-level matches were identified across the two 
raters for any individual examinee, the overall holistic rating 
of that individual was consulted. Finally, three TOPT item 
responses that were at the overall holistic rating for the 
examinee (or were all at the same level close to the holistic 
rating) were chosen for each individual from among the matches. 
These three item responses were to typify responses at that 
holistic level (or at the common level of the three item 
ratings) . For Spanish, when matches occurred there, three 
Advanced level items (generally including Picture Four — past 
tense narration) were used as examples from examinees identified 
as Intermediate; Picture Four and two Superior level items were 
used for examinees identified as Advanced, and three Superior 
items — most commonly supporting opinion (Topic 4), hypothesizing 
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(Topic 5), and giving a professional talk (Situation 4) — were 
used as examples for examinees identified as Superior. Examples 
were chosen for the French TOPT in a very similar fashion, except 
that the French TOPT included only two Superior level items. 
This meant that those same two items were repeatedly chosen as 
examples for the performance of Superior and Advanced level 
students. 

Each examinee was thus assigned a tentative rating — the 
rating of the three matched segments. In the case of Spanish, 
because of the high level of agreement between Blanco and 
Contreras, there was a single tentative rating for all but four 
of the 29 individuals included on the first roaster tape. For the 
French examinees, however, there was a single tentative level for 
only 10 of the 27 examinees on the first master tape. The others 
had a split tentative rating, the lower reflecting Manley's 
rating, the higher reflecting Huggin»s (e.g. IM/IH, A+/S) • In 
all of these cases, the two ratings differed by only one step on 
the ACTFL scale. 

Once three segments for each examinee were chosen, the order 
of examinees to appear on the first master tape was determined 
randomly by drawing each examinee's ID number at random from a 
hat. However, if two examinees of the same level were chosen in 
a row, the second was returned to the pile. In this way, ability 
levels appear in random order on the master tape, with no two 
ability levels presented in sequence. 

The first master tapes for French and Spanish were then 
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prepared by dubbing from the original examinee response tapes 
onto the master tape. 

7.1.4 Confirming the Master Tape Ratings 

In order to confirm that each examinee* s segment of three 
responses reflected the score assigned to it, CAL conducted a 
confirmatory study of the tentative ratings. David Hiple, 
director of the tester training program at ACTFL, submitted to 
CAL the names of the outstanding ACTFL raters and trainers for 
French and Spanish from across the country. Five for each 
language were contacted and agreed to participate in the study. 
Each was sent a copy of the master tape, a rating sheet, and 
instructions. (The names and affiliations of these raters, with 
a copy of the instructions and rating sheet, appear in Appendix 
N.) These raters were told to listen to each person and estimate 
his or her ACTFL rating based on the performances contained on 
the tape. These raters were NOT given the tentative rating of 
each segment. They worked totally independently and had about a 
week to accomplish their task. 
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7.1.5 Assigning "True Sc ores" to the Master Tape 

Once the raters 1 responses were received by GAL, their 
scoring data was entered into a computer. Their ratings for each 
examinee were examined together with the examinee's tentative 
rating (i.e., the agreed upon level (s) of the two Texas raters). 
When possible, a "true score" was assigned to each examinee. 
This "true score" was the level the examinee's responses on the 
first master tape was intended to typify. The complex process of 
assigning true scores from this data is described next. 

The logic behind this procedure was the desire to confirm 
the tentative Texas ACTFL ratings by the ratings of the five 
independent external raters. First, however, examinees for whom 
ratings were discrepant were eliminated from the pool of 
potential examinees to be included in the final master tape. 
Examinees were eliminated if there was no clear modal (average) 
rating with reference to the tentative score (e.g., 4:4,4,5,5,6 — 
bold type indicates tentative level 1 ) . The logic behind 
eliminating such examinees lies in the fact that the external 
raters were not in clear agreement and that there was no 
confirmation that the tentative rating lay within the boundaries 
of the external raters' rating. In the example above, the 
external raters as a group appear to place the examinee above the 



3 For this procedure, the lower of the French ratings 
(i.e., Manley's ratings) was used as the tentative rating since 
when in doubt, ACTFL raters are instructed to use the lower 
score. An additional reason for this decision was Manley's 
greater experience in rating tapes. 
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tentative score. 

Second, examinees were eliminated if a level split did not 
confirm the tentative rating. This happened when three raters 
assigned one level, while the tentative level and the other two 
raters were at the adjacent one (e.g., 3:3,3,4,4,4). This would 
indicate that the examinee is probably a borderline case, whose 
true score may well fall somewhere between the two levels. Using 
such an individual as an example of a level in the standard 
setting study would be confusing since he or she is not typical 
of any one level. 

Eliminating examinees according to the above guidelines 
reduced the number of examinees on the Spanish tape from 29 to 22 
and on the French tape from 27 to 19. Of those examinees 
retained, we assigned a true score using the following four 
methods : 



Method A 



In the first method, the true score was the 
tentative score confirmed by at least three of the 
five external raters (e.g. 6:5,6,6,6,7=6 / 
6:6,6,6,7,7=6 / 4:4,4,4,4,5=4 with the score after 
the = sign being the true score assigned to that 
person) . 50% of Spanish and 79%* of the French 



4 When Manley's lower ratings are used as the tentative 
score the figure is 68%. However, as noted above, for 13 of the 
19 examinees on the final tape, there was a level disagreement 
between Manley and Huggins. For the six examinees where Manley 
and Huggins agreed, four of the tentative scores were confirmed 
by the external raters with Method A and two with Method B. In 
the 13 cases where there was disagreement, in only one case were 
both Manley and Huggins* tentative rating discontinued (see 
Method C) . In the remaining 12 cases, 6 (50%) of Manley's lower 
tentative ratings were confirmed by Method A and one (8%) by 
Method B, while 5 (42%) of Huggins' higher tentative ratings were 
confirmed by this method. Thus, given the split tentative scores 
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examinees received true scores in this Banner. 

Method B 

In the second method (Method B) , the true score 
vas again the tentative score, confirmed by one or 
two of the five external raters and falling 
between the other scores. This confirmed that the 
tentative score was an average score (e.g. 
15:5, 6,6,7,7-6 / 3:2,2,3,3,4=3 / 2:1,1,2,3,3=2). 

The logic behind keeping a tentative score falling between 
the external rater's scores, even if it was not the modal 
score, is that on average, across ratings, that individual 
would receive the tentative score. Variation in assigned 
score for these people is considered to be due to rater 
error, and raters are equally likely to vary in either 
direction. 14% of the Spanish and 16% of the French true 
scores were assigned in this w? . 

Method C 

In methods C and D, the tentative score did not 
become the true score. In Method C, the "true 
score" was the score awarded by four or five of 
the five external raters even when different from 
the tentative score (e.g. 1:1,2,2,2,2=2). 22% of 
the Spanish examinees and 5% of the French 
examinees received true scores using this method 5 . 



Method D 

For two of the Spanish tapes (10%) and none of the 
French tapes, the true score was the modal score 
awarded by three of the five external raters when 

was the average of all the external rater's 
scores (3:3,4,4,4,5=4 / 2:2,3,3,3,4=3). 

The reason behind using methods C and D, in which the 
tentative score was not confirmed, is that samples of speech 
presented in three segments for any examinee may give the 
hearer a different impression when listened to by themselves 
t han when heard in the context of the entire tape, in this 

for French, 15 out of 19 tentative ratings (79%) were confirmed 
by Method A. 

5 If Manley's lower rating is taken to be the tentative 
score, then three additional tapes were awarded true scores in 
this manner. However, in these three cases, the true score was 
equal to Huggins* tentative score. 
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case, the reaction of the members of the standard setting 
committee would be more reflective of that of the five 
external raters who also heard only the selected segments 
than of the ACTFL Texas raters who h*ard the examinee's 
entire tape. Note that in Method C the true score was 
awarded by 80% of the external raters, indicating very high 
agreement, and that Method D, which involves a true score 
awarded by 60% of the raters, was used only for two Spanish 
tapes . 

Table 7.1 presents a frequency distribution of the methods 
used to award the true scores on the final Master Tape. 



Table 7.1 

Frequency of the Use of Four Methods to 
Award True Scores on the Master Tape 

Method Spanish Examinees French Examinees* 



A 50% 79% 

B 18% 16% 

C 22% 5% 

D 10% 0% 

Key: 

A Tentative score m true score, confirmed by three or more 
external raters 

B Tentative score ■ true score, confirmed by one or two external 
raters and falling midway between extreme ratings 

C Tentative score not confirmed; true score awarded by four or 
five external raters 

D Tentative score not confirmed? true score awarded by three of 
five external raters and falling between extreme scores 

* See footnotes 3 and 4 



Although originally five tapes estimated to be from each 
ACTFL level (except Intermediate Low) were sent to the Texas 
ACTFL-certif ied raters, because of the elimination of examinee 
examples and the assignment of actual true scores (rather than 
estimates based on the examinee's self-rating and the original 
judge's rating after trialing) , there was not an equal number of 
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examples from each ACTFL level on the final master tape for 
Spanish. All the French tapes were involved in the process, but 
many of them were eliminated, too. Below are the numbers of 
examples for each level appearing on the master tapes: 



ACTFL Level French Spanish 



Intermediate Low 0 0 

Intermediate Mid 6 3 

Intermediate High 3 5 

Advanced 4 5 

Advanced plus 3 2 

Superior 3 s 

High Superior o 1 



7.2 Setting t he Passing Standards 

The TEA submitted to CAL a list of teachers and teacher 
trainers from throughout Texas whom they deemed qualified to 
serve as judges on the standard setting committees. These 
individuals were then sent invitations by CAL to the standard 
setting sessions. 

The standard setting sessions for French and Spanish language 
teaching were held on Friday, October 19, in conjunction with the 
annual Texas Foreign Language Association fall convention held in 
Fort Worth. For bilingual education, the standard setting 
session was held on Thursday, November 1, at the civic Center in 
Lubbock, in conjunction with the annual Texas Association of 
Bilingual Educators fall conference. There were 16 judges on the 
French committee, 17 on the Spanish, and 13 on the bilingual 
education committee. Below are the names and affiliations of the 
members of each committee. 
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Members o f the French standard Setting Committee 



Ms. Peggy Beauvois-Hollon 

Nr. Louis A. Broussard 

Ms. Yvette J. De Jean 

Dr. Marion LeRoy Ellis 

Mr. Joe H. Galindo 

Ms. Carrie Harrington 

Dr. Michael 6. Hydak 

Ms. Sue E. Kimbro 

Dr. Cynthia Manley 

Ms. Sandra Miller 

Ms. Linda L. Nance 

Ms. Risa Pajestka 

Dr. Elaine M. Phillips 

Ms. Peggy Tharp 

Dr. Donald R. Vidrine 

Ms. Michele Wade 



University of Texas-Austin 
San Antonio ISD 
Houston ISD 
Lamar University 
El Paso ISD 
Fort Worth ISD 
Austin ISD 
Texarkana ISD 
Austin College 
Klein ISD 
San Antonio ISD 
Belton ISD 
Southwestern University 
San Angelo ISD 
University of North Texas 

Lubbock ISD 



Members of the Spanish Standard Setting Committee 



Ms. 


Suzanne Abbott 


Lubbock ISD 


Ms. 


Elizabeth Bailey 


Leander ISD 


Mr. 


Sam Calderdn 


Socorro ISD 


Ms. 


Rosario M. Cantii 


Northside ISD 


Dr. 


H. Eliot Chenaux 


Corpus Christi State University 


Ms. 


Kathy Hamm 


Dickinson ISD 


Ms. 


Elizabeth A. Haskins 


Edinburg ISD 


Ms. 


Nancy J. Lewis 


Abilene ISD 


Ms. 


Rachael A. Loman 


Donna ISD 


Ms. 


Olivia Munoz 


Houston ISD 


Dr. 


Manuel J. Ortufto 


Baylor University 


Ms. 


Rosemary Patterson 


Amarillo ISD 


Ms. 


Elizabeth Schacht 


Highland Park ISD 


Dr. 


Lorum H. Stratton 


Texas Tech University 


Ms. 


Christie Walker 


Warren ISD 


Ms. 


Blanca Barrera Watters 


White Settlement ISD 


Ms. 


Loretta Garcia Williams 


Piano ISD 
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Members of the Bilin gual Education Standard Setting Committee 



Ms. 


Carmen A. Dominguez 


Houston ISD 


Ms. 


Lucia E. Elizarde 


Harlingen ISD 


MS. 


Yolanda Espinoza 


San Marcos ISD 


Dr. 


Maria Loida Galvez 


Pasadena ISD 


Ms. 


Susana Gomez 


Lubbock ISD 


Ms. 


Joyce Hancock 


Lufkin ISD 


Dr. 


Roy Howard 


Texas Tech University 


Mr. 


Manuel A. Martinez 


Austin ISD 


Ms. 


Isabel 1 McLeod 


Amarillo ISD 


Ms. 


Elba-Maria Stell 


El Paso ISD 


Ms. 


Juanita villegas 


Lubbock ISD 


Dr. 


Judith Walker de Felix 


University of Houston 


Ms. 


Elsa Meza Zaragosa 


Corpus Christi ISD 



The task given to the members of the standard setting 



committees was to listen to each individual on the master tape 
and mark on a machine-readable response sheet whether or not they 
felt that person demonstrated enough speaking ability to be in a 
Texas classroom. The specific instructions that were read to the 
committee members appear in Appendix o. An example of the 
response sheet appears in Appendix P. 



7.3 Results o f the Standard Se tting Study 

Table 7.2 presents the make-up of the 16 members of the French 
Standard Setting Committee. 
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Table 7.2 

Descriptive Statistics on the French Standard 
Setting Committee Members 

hi Position 

Cumulative Cumulative 
POSITION Frequency Percent Frequency Percent 

Classroom Teacher 7 43.7 7 43.7 

Department Chair 3 18.8 10 62.5 

District Supervisor 1 6.2 11 68.7 

Teacher Trainer 5 31.3 16 100.0 
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B. Sex 

Cumulative Cumulative 
SEX Frequency Percent Frequency Percent 

Male 5 31.3 5 31.3 

Female 11 68.7 16 100. 0 

c T Ethnicity 

Cumulative Cumulative 
ETHNIC Frequency Percent Frequency Percent 

Hispanic 1 6.2 1 6.2 

White-NonHispan 13 81.2 14 87.5 

Black-NonHispan 2 12.5 16 100.0 

D. Region of Texas by Fir st Two D igits of Z ip Code 

Cumulative Cumulative 
REGION Frequency Percent Frequency Percent 

75 - Northeast 2 12.5 2 12.5 

76 - North Central 4 25.0 6 37.5 

77 - East 3 18.8 9 56.2 

78 - South Central 5 31.3 14 87.5 

79 - West 2 12.5 16 100.0 



Table 7.3 shows the percent of committee members rating 
examinees as acceptable at each ACTFL level. 



Table 7.3 

Mean Percent of Judges Rating Examinees 
Acceptable at Each Level (French Language) 

ACTFL Level Number of Examinees Mean Percentage 



Inter Mid 6 13.5 

Inter High 3 39.6 

Advanced 4 92.2 

Advanced Plus 3 100.0 

Superior 3 100.0 



Table 7.3 indicates that for the French standard setting 
committee members, an Intermediate High level performance was 
clearly not adequate, while an Advanced level performance was 
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deemed adequate on average by over 90% of the members. This 
suggests that the French standard should be Advanced. 

Table 7.4 presents the make-up of the 17 members of the 
Spanish Standard Setting Committee. 

Table 7.4 

Descriptive Statistics on the Spanish Standard 
Setting Committee Members 

A. Positi on 

Cumulative Cumulative 
POSITION Frequency Percent Frequency Percent 

Classroom Teacher 3 17.6 3 17.6 

Department Chair 8 47.1 11 64.7 

District Supervisor 3 17.6 14 82.4 

Teacher Trainer 3 17.6 17 100.0 

Pr Sex 

Cumulative Cumulative 
SEX Frequency Percent Frequency Fercent 

Male 4 23.5 4 23.5 

Female 13 76.5 17 100.0 



C_« Ethnicity 

Cumulative Cumulative 
ETHNIC Frequency Percent Frequency Percent 



Hispanic 9 52.9 9 52.9 

White-Nonhispan 8 47.1 17 100.0 
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D. Region of Tex as bv First Two Digits of Zip Code 

Cumulative Cumulative 

REGION Frequency Percent Frequency Percent 

75 - Northeast 2 11.8 2 11.8 

76 - North Central 2 11.8 4 23.5 

77 - East 3 17.6 7 41.2 

78 - South Central 5 29.4 12 70.6 

79 - West 5 29.4 17 100.0 



Table 7.5 shows the percent of Spanish language committee 
members rating examinees as acceptable at each ACTFL level. 

Table 7.5 

Mean Percent of Judges Rating Examinees 
Acceptable at Each Level (Spanish Language) 

ACTFL Level Number of Examinees Mean Percentage 



later Mid 3 25.5 

Inter High 5 25.9 

Advanced 5 77.7 

Advanced Plus 2 94.1 

Superior 6 94.1 

High Superior 1 100.0 



Table 7.5 indicates that for the Spanish standard setting 
committee members, Intermediate level performances were clearly 
not adequate, while the Advanced level performance was deemed 
adequate on average by over 75% of the members. This suggests 
that the Advanced level should be the standard. 

Table 7.6 presents the make-up of the 13 members of the 
Bilingual Education Standard Setting Committee. 
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Table 7.6 

Descriptive Statistics on the Bilingual Education 
Standard Setting Committee Members 

Kx Position 

Cumulative Cumulative 
POSITION Frequency Percent Frequency Percent 

Classroom Teacher 10 76.9 10 76.9 

District Supervisor 1 7.7 11 84.6 

Teacher Trainer 2 15.4 13 100.0 

£j S_£X 

Cumulative Cumulative 
SEX Frequency Percent Frequency Percent 

Male 2 15.4 2 15.4 

Female 11 84.6 13 100.0 

C. Ethnicity 

Cumulative Cumulative 
ETHNIC Frequency Percent Frequency Percent 

Hispanic 10 76.9 10 76.9 

White-NonHispan 3 23.1 13 100.0 

D. Region of Texas bv First Two Digits of Zip Code 

Cumulative Cumulative 
REGION Frequency Percent Frequency Percent 

75 - Northeast 1 7.7 1 7.7 

77 - East 3 23.1 4 30.8 

78 - South Central 4 30.8 8 61.5 

79 - West 5 38.5 13 100.0 



Table 7.7 shows the percent of bilingual education committee 
members rating examinees as acceptable at each ACTFL level. 
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Table 7.7 

Mean Percent of Judges Rating Examinees 
Acceptable at Each Level (Bilingual Education) 



ACTFL Level 



Number of Examinees 



Mean Percentage 



Inter Mid 
Inter High 
Advanced 
Advanced Plus 
Superior 
High Superior 



3 
5 
5 
2 
6 
1 



12.8 
21.5 
83.1 
88.5 
96.2 
100.0 



Table 7.7 indicates that, as for French and Spanish, there 
is a clear dividing line between Intermediate High and Advanced. 
For the bilingual education committee members, an Intermediate 
High level performance was clearly not adequate, while the 
Advanced level performance was deemed adequate on average by over 
80% of the committee members. 

The above tables show that for each group there is a clear 
line dividing performances at the Intermediate High and Advanced 
levels. These figures support the recommendation of Advanced as 
the passing score fcr all three groups. 
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APPENDIX A 



JOB-RELATEDNESS SURVEY; 
INSTRUCTIONS AND 
MACHINE READABLE RESPONSE SHEET 
(Fo/ Each Group of Teachers) 



Texas Oral Proficiency Test (TOFT) 
French Language Teachers 

JOB KELATTEDNESS SURVEY 

RETURN BY MAY 4, 1990 

PRODUCTION 

The Texas Education Agency is developing a test of oral proficiency in French which 
will be required of individuals seeking secondary certification as a French language 
teacher or an elementary specialization in French. The Texas Oral Proficiency Test 
French (TOPT-French) will be a tape-mediated test From a master tape and via a 
test booklet, examinees will be presented with approximately twenty speaking tasks. 
These tasks wfll allow them to demonstrate their ability to speak French. Successful 
performance of these tasks requires various levels of French speaking ability; some are 
fairly easy to perform, while others are considerably more challenging. The examinees* 
responses will be recorded on examinee response tapes. After examinees complete the 
test, their performance, as recorded on the tapes, will be scored by trained raters. 

This survey presents you with 38 speaking tasks, such as may appear on the TOFT- 
French. For each task, you are to indicate whether, in your professional opinion, 
French language teachers need to have the ABILITY to carry out this task in order to 
perform successfully in French language classrooms in the state of Texas. Note that 
the question « not whether French language teachers need to cany out the task in the 
classroom, but whether French language teachers need the level of ability necessary to 
carry out the task. 

You are one of a sample of Texas French language teachers selected to receive this 
survey. The results will assist the TEA in determining the level of speaking skills in 
French needed by French language teachers in Texas. Your responses are important 
and your assistance to the TEA is appreciated. 



DIRECTIONS 

Your survey packet contains: this survey booklet, a blue and white machine-readable 
survey response sheet, and a stamped, pre-addressed return envelope. Note that data 
for this survey are being collected with machine-readable response sheets. Please do 
not fold the survey response sheets. 

There are five steps to completing this survey. Follow all directions carefully and use a 
No. 2 pencil. It is estimated that this survey will require 15 to 20 minutes to complete. 
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STEP 1 ID NUMBER 



Please write your social security number in the boxes in the area entitled ID 
NUMBER on the top left-hand corner of the machine-readable survey response 
sheet Then fill in the circle corresponding to the number m each box* NOTE: 
Your soda! security number win only be used for data pro ces sin g purposes and 
will not be used to identify any individual respondent so mis survey. 

EXAMPLE 

This is what your response sheet would look Eke if your social security number were 
123*454789: 
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STEP 2 



DEMOGRAPHIC INFORMATION 



For demographic purposes, please answer each lettered question presented on 
the next page in the box labeled DEMOGRAPHIC INFORMATION. Write 
your answer in the area entitled SPECIAL CODES on the top left-hand corner 
of the response -sheet For ecch lettered question (A through 0% write the 
number of your answer in the block on the answer sheet Then fill in the circle 
corresponding to the number of your answer. 

EXAMPLE 

This is what your response sheet would look Vke if you were a high school teacher 
(Question A) with a secondary French certificate (Question B) and between 3 and 5 
years of experience (Question C), etcj 
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DEMOGRAPHIC INFORMATION 



A. What U your current level of assignment? 



(0) Elementary 

(1) Junior High or Middle School 



(2) High School 

(3) Other 



B. What certificate or endorsement do job hold? 

(0) Elementary certificate with French specialization 

(1) Secondary French certificate 

(2) Both of the above 

(3) None of the above 

C Cow many years of French language teaching experience do yon have? 



(0) 1-2 years 

(1) 3-5 years 

(2) 6-10 years 



(3) 11-15 years 

(4) 16-19 years 

(5) 20 or more years 



D. What levels of French language classes have you taught during 
the past three years? 

(0) All beginning (first and second year) classes 

(1) Mostly beginning classes, some advanced (third year and beyond) 

classes 

(2) About half beginning, half advanced classes 

(3) Mostly advanced classes, some beginning classes 

(4) All advanced classes 



E. What is the highest degree that yon hold? 



(0) No degree 

(1) Bachelor's 

F. What is your ethnic group? 

(0) Hispanic 

(1) Black 

G. Whai Is your sex? 
(0) M^e 



(2) Master's 

(3) Doctorate 



(2) White 

(3) Other 



(1) Female 
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STEP 3 



RESPONSES TO SPEAKING TASKS 



Listed on the survey response sheet Is a series of speaking tasks requiring 
various degrees of language ability to perform. For each task, indicate whether, 
in your professional opinion, French language teachers need to have the 
language ability necessary to carry out the task in order to perform successfully 
in the French language classroom. In other words, for each task, ask yourself: 



Is the level of ability 
required to perform this task 
needed by French language teachers 
in Texas public schools? 



Important The question is NOT "Do French teachers need to cany out this 
task in the classroom?" Rather, the question is "Do French language teachers 
need to have the French language ability to carry out this task?" 

Fill in the letter that represents your response to this question m the 
appropriate column on the response sheet. The columns are as follows: 

A - Definitely Yes 

B « Probably Yes 

C - Maybe 

D m Probably No 

E « Definitely No 

Following the examples below are detailed descriptions of the speaking tasks. 
Be sure to read them before making your response. 

EXAMPLES 

Here are two example tasks with responses completed for yow 
Example A 

Extend an Invitation 

Be able to politely invite someone to your home for a party or other social 
function. 

// in your opinion, French languor ieacfiers should dtfmttily have the Itvtf of ot»li*y 
required to perform this speaking ~isk (independent of whedier they would need to do the 
task in the classroom), tlxen you would darken circle "A" in l)ie first column of the 
response sheet 



Example B 



Negotiate Renting Temporary Living Quarters 

Be able to negotiate ft rental agreement with a landlord; ask questions about 
what is included in the rent, and ask for clarification of the rental agreement 

If, in your opinion, French language teachers should probably have the Uwl «f obtlitv 
required to perform this speaking task (independent of whether they would need to do the 
task in tlie classroom), tlusn you would darken circle Tin the second column of the 
response sheet. 

If you made the above two responses to the example tasks, your survey response ''set 
would look like this: 



GENERAL PURPOSE DATA SHEET H 
form no. 70921 



Definitely So * — | 



Probably No 



Kaybe 



Probably T«» 



&ef laltely Tea 



A. Extend an Invitation , „ . . 

B. Negotiate Renting Temporary Living Quartern- 
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Now please make your response for each of the 38 speaking tasks listed on the 
following pages on the appropriate line of the survey response sheet 
Remember to ask yourself, for each task: 



Is the level of ability 
required to perform this t&»ic 
needed by French language teachers 
in Texas public schools? 
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SPEAKING TASKS 

1. Introduce Yourself 

Be able to give your name and basic personal information such as would be 
given at a first meeting. 

2. Explain a Familiar. Simple Process 

Be able to explain how to accomplish everyday processes wch as writing a 
check, borrowing a book from the library, or taking attendance in the classroom. 

3. Describe a Sequence of Events in the Past 

Be able to use and sequence language indicating past time in order to narrate 
an event or incident which occurred recently. 

4. Propose and Defend a Co urse of Action with Persuasion 

In light of at least two possible choices of action, be able to propose and defend 
a course of action in such a way as to persuade others to accept your choice. 

5. Pcscribe Typtol Routines, 

Be able to use and sequence language indicating present or habitual time in 
order to narrate recurring events or routines, everyday activities, etc 

6. Make PvTchaseg 

Be able to request hems, discuss prices, and handle currency in a situation 
involving a purchase. 

7. Talk About Personal Activities 

Be able to talk about your leisure activities, favorite pastimes, and preferred 
hobbies. 

8. Hypothesize About an Im personal Topic 

Be able to discuss various possibilities ("what if situations) surrounding an 
abstract, impersonal topic. 

(FOR SURVEY PURPOSES ONLY) 
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9. Talk About Family Members 

Be able to give the names of the members of your family and simple descriptive 
information, such as their occupations and physical characteristics. 

la Give a Brief. Organized Factual Summary 

Be able to summarize in an "oral report* 1 fashion factual information about 
topics of a personal or professional nature. 

U. State Your Personal Point of View on a C ontroversial Subject 

Be able to state what you believe on a controversial subject and why you bold 
those beliefs. 

12. Describe Expected Future Events 

Be able to use and sequence language indicating fir ire time in order to narrate 
expected occurrences of a personal nature, such as a planned trip or activity. 

13. Explain a Complex Process in Detail 

Be able to explain in detail a non-routine process of an impersonal nature, such 
as how to carry out a scientific investigation or how to write a term paper. 

14. Procr a Mefll 

Be able to ask questions about menu hems, order food, and ask for and settle a 
ML 

15. Express Per sonal Apologies 

Be able to apologize clearly and appropriately to an offended party. 

16. Give Advice 

Be able to give advice to someone faced with making a decision between two or 
more choices, giving supporting reasons for the advice given. 

17. Hypothesize About a Personal Situation 

Be able to say what you would do in a hypothetical situation. 

[FOR SURVEY PURPOSES ONLY] 
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la Describe Your Daily Routine 

Be able to nai rate your typical dally activities. 

39. Give Instructions 

Be able to give instructions and explain the steps involved in carrying out an 
activity. 

20. Qjve p Brief Persona 1 Hiaoiy 

Be able to talk about your personal background. 

21. State Advantages and Disa dvantages 

Be able to state the advantages and disadvantages of a situation (such as living 
in a big dry), a decision (such as going to college), or an object that has 
affected society (such as the computer). 

22. Support Opinions 

Be able to state, support and defend a personally-held opinion or belief about 
an issue. 

23. Describe Health Problems 

B- able to describe health problems or conditions, 

24. Discuss a Professional Topic 

Be able to discuss at length and in detail a topic of professional interest 

25. Describe a Complex Object in Detail 

Be able to describe a complex object such as a car or bicycle in detail and with 
precise vocabulary. 

26. Lod g e a Complaint 

Be able to lodge a complaint, giving the reasons for and details behind the 
complaint. 

[FOR SURVEY PURPOSES ONLY] 
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27. Talk About Your Future Plans 

Be able to state and describe your personal or professional plans, goals and 
ambitions. 

2a Gfve a Professional Talk 

Be able to present a talk on a topic of professional interest. 

29. Make Arrangements for Future Activities 

Be able to inquire about and to make arrangements for future activities, and to 
set the date, time and place. 

3a Evaluate Issu es Surrounding a Conflict 

Be able to present arguments on both sides of a familiar issue or topic and 
evaluate their relative merits. 

31. Give Directions 

Be able to give directions on how to get from one place to another. 

31 Describe 9 Place 

Be able to describe in detail a particular place, such as a school, a store, or a 
park. 

33. Explain a Complex Process of a Personal Nature 

Be able to describe and explain in detail a non-routine process such as how to 
get a job, or how to apply to cohege. 

34. Hypothesize About Probable Outcomes 

Be able to discuss what could happen if something unexpected occurs. 

35. Correct an U nexpected Situation 

Be able to handle an unexpected outcome, such as receiving faulty merchandise. 

[FOR SURVEY PURPOSES ONLY] 
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36. Change Someone*! Behavior through Persuasion 

Be able to persuade someone to do something he or she » not inclined to do, 
or to cease doing something which is annoying to you. 

37. Describe Habitual Actfens in the Past 

Be able to describe people, places or things in the past, such as the work 
schedule you used to have or leisure activities you used to da 

3& Compare and Contrast Tw o Objects or Place 

Be able to compare and contrast two objects, places, or customs. 



STEP 4 ADDITIONAL COMMENTS 

Please use the space provided in the three WRITE-IN AREAS on the back of 
the survey response sheet for any additional comments you wish to make 
regarding the oral language functions to be included on the TOFT-French. 



STEP 5 RETURNING THE SURVEY 

Unfold the enclosed pre-addressed, stamped envelope. Insert the blue and 
white machine-readable survey response sheet into the envelope, being careful 
not to fold iL Return the machine-readable survey response sheet only as soon 
as possible, but postmarked no later than MAY 4, 1990, to: 

Mr. Dorry Kenyon 
Center for Applied linguistics 
HIS 22nd Street, NW 
Washington, DC 20037 



Thank you for your participation in this survey. 



RETURN BY MAY 4, 1990 
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SPECIAL CODES 
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TOFT Job-Re latednesa Survey 
French Language Teaching 




Definitely Ho 
Frobably No 



GENERAL PURPOSE DATA SHEET II 
form no. 70921 
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Frobably Yes 



^ Definitely Tea 



■> 



1. Introduce Yourself---- « ^--~--i-----__---___->^0 

2. Explain a Familiar, sinple Process > © 

3. Describe a Sequence of Events in the Past — « > 0 

4. Propose ft Defend a Course of Action with Persuasion> © 

5. Describe Typical Routines -> 0 

6. Make Purely ses— — — > 0 

7. Talk About Personal Activities > © 

8. Hypothesize About an Impersonal Topic > © 

9. Talk About Family Members > © 

10. Give a Brief, Organized Factual Summary — > 0 

11. State Personal Point of View (Controversial Subject) 

12. Describe Expected Future Events— > 

13. Explain a Complex Process in Detail — > 

14 . Order a Meal — — — 

15. Express Personal Apologies 

16. Give Advice——*—— — - > 

17. Hypothesize About a Personal Situation > 

18. Describe Your Daily Routine — > 

19. Give Instructions > 

20. Give a Brief Personal History > 

21. State Advantages and Disadvantages > 

22. Support Opinions — — > 

23. Describe Health Problems > 

24. Discuss a Professional Topic > 

25. Describe a Complex Object in Detail > 

26. Lodge a Complaint— > 

27. Talk About Your Future Plans > 

28. Give a Professional Talk > 

29. Make Arrangements for Future Activities— > 

30. Evaluate Issues Surrounding a Conflict* — > 

31. Give Directions > 

32. Describe a Place— — - — . > 

33. Explain a Complex Process of a Personal Nature > 

34. Hypothesize About Probable Outcomes > 

35. Correct an Unexpected Situation > _ 

36. Change Someone* s Behavior through Persuasion > © 

37. Describe Habitual Actions in the Past > © 

38. Compare and Contrast Two Objects or Places — — > 
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Texas Oral Proficiency Test (TOFT) 
Spanish Language Teachers 



JOB-RELATEDNESS SURVEY 
RETURN BY MAY 4, 1990 

INTRODUCTIO N 

The Texas Education Agency is developing a test of oral proficiency in Spanish which 
will be required of individuals seeking secondary certification as a Spanish language 
teacher or an elementary specialization in Spanish. The Texas Oral Proficiency Test in 
Spanish (TOPTSpanish) will be a tape-mediated test From a master tape and via a 
test booklet, examinees wilJ be presented with approximately twenty speaking tasks. 
These tasks will allow them to demonstrate their ability to speak Spanish. Successful 
performance of these tasks requires various levels of Spanish speaking ability; some are 
Cairly easy to perform, while others are considerably more challenging. The examinees' 
responses will be recorded on examinee response tapes. After examinees complete the 
test, their performance, as recorded on the tapes, will be scored by trained raters. 

This survey presents you with 38 speaking tasks, such as may appear on the TOPT- 
Spanish For each task, you are to indicate whether, in your professional opinion, 
Spanish language teachers need to have the ABILITY to carry out this task in order to 
perform successfully in Spanish language classrooms in the state of Texas. Note that 
the question is not whether Spanish language teachers need to carry out the task in the 
classroom, but whether Spanish language teachers need the level of ability necessary to 
carry out the task 

You are one of a sample of Texas Spanish language teachers selected to receive this 
survey. The results will assist the TEA in determining the level of speaking skills in 
Spanish needed by Spanish language teachers in Texas. Your responses are important 
and your assistance to the TEA is appreciated. 

DIRECTIONS 

Your survey packet contains: this survey booklet, a blue and white machine-readable 
survey response sheet, and a stamped, pre-addressed return envelope. Note that data 
for this survey are being collected with machine-readable response sheets. Please do 
not fold the survey response sheets. 

There are five steps to completing this survey. Follow all directions carefully and use a 
No. 2 pencil. It is estimated that this survey will require 15 to 20 minutes to complete. 
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STEP 1 



ID NUMBER 



Please write your social security Dumber in the boxes in the area emitted ID 
NUMBER on the top left-hand corner of the machine-readable survey response 
sheet Then fin in the circle corresponding to the number in each boat NOTE: 
Your social security number will only be used for data processing purposes and 
wQ] not be used to identify any individual respondent to this survey. 



EXAMPLE 

This t what your response sheet would took Eke if your social security number were 
123-454789: 
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STEP 2 



DEMOGRAPHIC INFORMATION 



For demographic purposes, please answer each lettered question presented on 
the next page m the box labeled DEMOGRAPHIC INFORMATION. Write 
your answer in the area entitled SPECIAL CODES on the top left-hand corner 
of the response sheet For each lettered question (A through G), write the 
number of your answer in the block on the answer sheet Then fill in the circle 
corresponding to the number of your answer. 



EXAMPLE 

This is wiuu your response sheet would look Eke if you were a high school teacher 
(Question A) with a secondary Spanish certificate (Question B) and between 3 and 
5 years of experience (Question C), etc 
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DEMOGRAPHIC INFORMATION 



What Is your current level of assignment? 



(0) 
0) 



Elementary 

Junior Hijjh or Middle School 



(2) High School 

(3) Other 



B. What certificate or endorsement do yon bold? 

(0) Elementary certificate with Spanish specialization 

(1) Secondary Spanish certificate 

(2) Both of the above 

(3) None of the above 

C Bow many years of Spanish language teaching experience do you have? 

(0) 1-2 years (3) 11-15 years 

(2) 3*5 years (4) 16-19 years 

(2) 6-10 years (5) 20 or more years 

D. What levels of Spanish language classes have yoo taught during 
the past three years? 

(0) All beginning (first and second year) classes 

(1) Mostly beginning classes, some advanced (third year and beyond) 

classes 

(2) About half beginning, half advanced classes 

(3) Mostly advanced classes, some beginning classes 

(4) All advanced classes 

E. What Is the highest degree that yoo bold? 

(0) No degree (2) Master's 

(1) Bachelor's (3) Doctorate 

F. What is your ethnic group? 

(0) Hispanic (2) White 

(1) Black (3) Other 

G. What Is your sex? 



(0) 



Male 



Female 
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STEP 3 



RESPONSES TO SPEAKING TASKS 



Listed cm the survey response sheet is a series of speaking tasks requiring 
various degrees of language ability to perform. For each task, indicate whether, 
in your professional opinion, Spanish language teachers need to have the 
language ability necessary to carry out the task in order to perform successfully 
in the Spanish language classroom. In other words, for each task, ask yourself: 



Is the level of ability 
required to perform this task 
needed by Spanish language teachers 
in Texas public schools? 



Important The question is NOT Do Spanish teachers need to carry out this 
task in the classroom?" Rather, the question is "Do Spanish language teachers 
need to have the Spanish language ability to carry out this task?" 

FQl in the letter that represents your response to this question in the 
appropriate column on the response sheet The columns are as follows: 

A «■ Definitely Yes 

B - Probably Yes 

C » Maybe 

D - Probably No 

E ■ Definitely No 

Following the examples below are detailed descriptions of the speaking tasks. 
Be sure to read them before making your response. 

EXAMPLES 

Here are two example tasks wilJi responses completed for you: 
Example A 

Extend j>n Invitation 

Be able to politely invite someone to your home for a party or other social 
function. 

// in your opinion, Spanish language teacliers shoull dtfmiftly have the Uvl of ability 
required to perform this speaking task (independent of whedier they would need to do the 
task in the classroom), tlum you would darken circle VT in die first column of the 
response sheet 

4 



Example B 



Negotiate Renting Temporary Living Quarters 

Be able to negotiate a rental agreement with a landlord, ask questions about 
what is included in the rent, and ask for clarification of the rental agreement 

If, in your opinion, Spanish language teachers should probably have she !•»•! a! obllity 
required to perform this speaking task (independent of whether they would need to do the 
task in the classroom), then you would darken circle m BT in the second column of the 
response sheet 

If you made the above two responses to the example tasks, your survey response sheet 
would look like this: 



Definitely So i 



GENERAL PURPOSE DATA SHEET 0 
form no. 70921 



Probably Wo 



Maybe 



Probably Tee 



Be finitely fee 
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A. Extend en Invitation---- - -- ■■■ ■ --■ 

B. Negotiate Eeotlog Temporary Living Quarters- 
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Now please make your response for each of the 38 speaking tasks listed on the 
following pages on the appropriate line of the survey response sheet 
Remember to ask yourself, for each task 



Is the level of ability 
required to perform this task 
needed by Spanish language teachers 
in Texas public schools? 
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SPEAKING TASKS 

1. Introduce Yourself 

Be able to gfve your name and bask personal information aucb as would be 
given at a first meeting. 

Z Explain a Familiar. Simple Process 

Be able to explain bow to accomplish everyday processes such as writing a 
check, borrowing a book from the library, or taking attendance in the classroom 

3. Describe a Sequence of Events in the Past 

Be able to use and sequence language indicating past time in order to narrate 
an event or incident which occurred recently. 

4. Propose ami Defend a Course of Action with Persuasion 

In 'jght of at least two possible choices of action, be able to propose and defend 
a course of action in such a way as to persuade others to accept your choice. 

5. Describe Typical Routines, 

Be able to use and sequence language indicating present or habitual time in 
order to narrate recurring events or routines, everyday activities, etc 

& Mate Purchases 

Be able to request hems, discuss prices, and handle currency in a situation 
involving a purchase. 

7. Talk About Personal Activities 

Be able to talk about your leisure activities, favorite pastimes, and preferred 
hobbies. 

& Hypothesis Afrovt an Impersonal Topic 

Be able to discuss various possibilities (Vhat if situations) surrounding an 
abstract, impersonal topic 

[FOR SURVEY PURPOSES ONLY] 
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9. Talk Ateaa Family Member? 

Be able to give the names of the members of your family and simple descriptive 
information, such as their occupations and physical characteristics. 

10. gig aJBrjel Organizes? Eacmal Summag 

Be able to summarize in an "oral report* fashion factual information about 
topics of a personal or professional nature. 

11. Stale Your Pe rsonal Point of View on a Controversial Subject 

Be able to state what you believe on a controversial subject and why you hold 
those beliefs. 

12. Describe Expected Future Events 

Be able to use and sequence language indicating future time in order to narrate 
expected occurrences of a personal nature, such as a planned trip or activity. 

n. Eqflain a Complex Process in DaaD 

Be able to explain in detail a non-routine process of an impersonal nature, such 
as how to carry out a scientific investigation or how to write a term paper. 

14. Order a Meal 

Be able to ask questions about menu hems, order food, and ask for and settle a 
bOL 

15. Express Personal Apologies 

Be able to apologize clearly and appropriately to an offended party. 

16. Qfvc Ac^'cg 

Be able to give advice to someone raced with making a decision between two or 
more choices, giving supporting reasons for the advice given. 

17. Hypothesize About a Persona) Situation 

Be able to say what you would do in a hypothetical situation. 

[FOR SURVEY PURPOSES ONLY] 
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la Describe Your Daily Routine 

Be able to narrate your typical daOy activities. 

19. Give Instrvrtons 

Be able to give instructions and explain the steps involved in carrying out an 
activity. 

20. pjvc a Brief Personal History 

Be able to talk about your personal background 

21. State Advantages and Disadvantages 

Be able to state the advantages and disadvantages of a situation (such as Irving 
in a big dry), a decision (such as going to college), or an object that has 
affected society (such as the computer). 

22. Sypppp QpioiflPi 

Be able to state, support and defend a personally-held opinion or belief about 
an issue. 

23. Describe Health Problems 

Be able to describe health problems or conditions. 

24. Discuss g Profession?! Topic 

Be able to discuss at length and in detail a topic of professional interest 

25. Describe a Complex Object in Detail 

Be able to describe a complex object such as a car or bicycle in detail and with 
precise vocabulary. 

26. Lodge a Complaint 

Be able to lodge a complaint, giving the reasons for and details behind the 
complaint 

[FOR SURVEY PURPOSES ONLY] 
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27. Talk About Your Futtir* 

Bc ^ to dcscrib c your personal or professional plans, goals and 

MDDlOOnS. 

28. Give a Profession] Tfllft 

Be able to present a talk on a topic of professional interest 

29. Mate Arrangements for Fut ure Activitj fl 

Be able to inquire about and to make arrangements for future activities, and to 
set the date, time and place, 

30. Evaluate Issues Surrounding f {fa^fr 

Be able to present arguments on both sides of a familiar issue or topic and 
evaluate their relative merits. 

31. Give Direction s 

Be able to give directions on how to get from one place to another. 

32. Describe a Plqog 

Be able to escribe in detail a particular place, such as a school, a store, or a 

33 - E*Ptein a Complex JW* < o f a p erS nn a | f tf^ 

Be able to describe and explain in detail a ixm-routine process such as how to 
get a job, or bow to apply to college, 

34- HvPO^esize About Prohahly fHflTOff 

Be able to discuss what could happen if something unexpected occurs, 
35. Correct an Unexpected S fty?tf?n 

Be able to handle an unexpected outcome, such as receiving faulty merchandise. 

[FOR SURVEY PURPOSES ONLY] 
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36. Change Some one's Behavior through Persuasion 

Be able to persuade someone to do something he or she is not inclined to do, or 
to cease doing something which is annoying to you. 

37. Describe Habitual Actions in the Past 

Be able to describe people, places or things in the past, such as the work 
schedule you used to have or leisure activities you used to do. 

38. Compare and Contrast Two Objects or Places 

Be able to compare and contrast two objects, places, or customs. 



STEP 4 ADDITIONAL COMMENTS 

Please use the space provided in the three WRITE-IN AREAS on the back of 
the survey response sheet for any additional comments you wish to make 
regarding the oral language functions to be included on the TOPT-Spanish. 



STEP $ RETURNING THE SURVFV 

Unfold the enclosed pre-addressed, stamped envelope. Insert the blue and white 
machine-readable survey response sheet into the envelope, being careful not to 
fold iL Return the machine-readable survey response sheet only as soon as 
possible, but postmarked no later than MAY 4, 1990, to: 

Mr. Dorry Kenyan 
Cente. for Applied Linguistics 
1118 22nd Street, NW 
Washinron, DC 20037 



Thank you for your participation in this survey. 



RETURN BY MAY 4, 1990 
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GENERAL PURPOSE DATA SHEET It 
form no. 7092 1 



Maybe 



c — 



Probably Tea 
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Definitely Tea 



1 . Introduce your»eX£---------------------------------> 

2. Explain a Familiar , Simple Process- — - — > 

3. Describe a Sequence of Events in the J ast > 

4. Propose a Defend a Course of Action with Persuasion> 

5. Describe Typical Routines- — - — > 

6. Make Purchases— •> 

7. Talk About Personal Activities™ > 

8. Hypothesize About an Impersonal Topic > 

9. Talk About Family Members — > 

10. Give a Brief , Organized Factual Summary- > 

11. state Personal Point of View (Controversial Subject) 

12. Describe Expected Future Events > 

13. Explain a Complex Process in Detail——-— — — > 

14. Order a Meal- — — — > 

15. Express Personal Apologies— — > 

16. Give Advice — - — > 

17. Hypothesize About a Personal Situation— — - — — > 

18. Describe Your Daily Routine — — — > 

19. Give Instructions™ — - — — — > 

20. Give a Brief Personal History -> 

21. State Advantages and Disadvantages—- > 

22. Support Opinions — -> 

23. Describe Health Problems — — — > 

24. Discuss a Professional Topic———- > 

25. Describe a Complex Object in Detail—- — > 

26. Lodge a Complaint- — -> 

27. Talk About Your Future Plans— ——-———> 

28. Give a Professional Talk——————— — - > 

29. Make Arrangements for Future Activities-— 

30. Evaluate Issues Surrounding a Conflict— ————> 

31. Give Directions——— — — — - — > 

32. Describe a Place — — > 

33. Explain a Complex Process of a Personal Nature > 

34. Hypothesize About Probable Outcomes > 

35. Correct an Unexpected Situation—— — - — — - — > 

36. Change Someone's Behavior through Persuasion— — > 

37. Describe Habitual Actions in the Past — — — — -> 

38. Compare and Contrast Two Objects or Places— — — > 
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Texas Oral Proficiency Test (TOFT) 
Bilingual Education Teachers 

JOB-RELATEDNESS SURVEY 

RETURN BY MAY 4, 1990 

INTRODUCTION 

The Texas Education Agency is developing a test erf oral proficiency in Spanish which 
wOJ be required of individuals seeking a certificate or an endorsement for bilingual 
education. The Texas Oral Proficiency Test in Spanish (TOPT-Spanish) will be a tape- 
mediated test From a master tape and via a test booklet, examinees will be presented 
with approximately twenty speaking tasks. These tasks will allow them to demonstrate 
their ability to speak Spanish. Successful performance of these tasks requires various 
levels of Spanish speaking ability; some are fairly easy to perform, while others are 
considerably more challenging. The examinees* responses will be recorded on examinee 
response tapes. After examiners complete the test, their performance, as recorded on 
the tapes, will be scored by trained raters. 

This survey presents you with 38 speaking tasks, such as may appear on the TOPT- 
Spanish. For each task, you are to indicate whether, in your professional opinion, 
bilingual education teachers need to have the ABILITY to carry out this task in order 
to perform successfully in bilingual education classrooms in the state of Texas. Note 
that the question is not whether bilingual education teachers need to carry out the task 
in the classroom, but whether bilingual education teachers need the level of ability 
necessary to carry out the task. 

You are one of a sample of Texas bilingual education teachers selected to receive this 
survey. The results will assist the TEA in determining the level of speaking skills in 
Spanish needed by bilingual education teachers in Texas. Your responses are 
important and your assistance to the TEA is appreciated 



DIRECTIONS 

Your survey packet contains: this survey booklet, a blue and white machine-readable 
survey response sheet, and a stamped, pre-addressed return envelope. Note that data 
for this survey are being collected with machine-readable response sheets. Please do 
not fold the survey response sheets. 

There are five steps to completing this survey. FoDow all directions carefully and use a 
No. 2 pencil. It is estimated that this survey wiD require 15 to 20 minutes to complete. 



1 
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STEP I 



00 NUMBER 



Please write your social security number in the bases in the area entitled ID 
NUMBER on the top left-hand corner of the machine-readable survey response 
sheet Then fin in the circle corresponding to the number in each box NOTE: 
Your soda! security number wiD only be used for data processing purposes and 
will not be used to identify any individual respondent to this survey. 

EXAMPLE 

This is what your response sheet would look Eke |T your scool security number were 
123-45-6789: 
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STEP 2 DEMOGRAPHIC INFORMATION 

For demographic purposes, please answer each lettered question presented on 
the next page in the box labeled DEMOGRAPHIC INFORMATION. Write 
your answer in the area entitled SPECIAL CODES on the top left-hand corner 
of the response sheet For each lettered question (A through G), write the 
number of your answer in the block on the answer sheet Then fill in the circle 
corresponding to the number of your answer. 

EXAMPLE 

This is what your response sfieet would look like $you were an elementary school 
teacher (Question A) with a certificate in bilingual education (Question B) and 
between 3 and 5 years of experience (Question Q, etc: 
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DEMOGRAPHIC INFORMATION 
A. What Is your current level of assignment? 



(0) Elementary 

(1) Junior High or MiddJe School 



(2) High School 

(3) Other 



B. Do you bold a certificate or endorsement In bilingual education? 

(0) Yes 

(1) No 

C How many years of bilingual education teaching experience do you have? 



(0) 1-2 yean 

(1) 3-5 yean 

(2) 6-10 yean 



(3) 11-15 yean 

(4) 16-19 yean 

(5) 20 or more years 



D. What levels of bilingual classes have you taught during the past 
three years? (select only one) 

(0) Early Childhood 

(1) Grades 1-3 

(2) Grades 4-6 

E. What is the highest degree that you hold? 



(0) No degree 

(1) Bachelor's 

F. What is your ethnic group? 

(0) Hispanic 

(1) Black 

G. What is your sex? 



(2) Masters 

(3) Doctorate 



(2) White 

(3) Other 



(0) Male 



(1) Female 
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STEP 3 



RESPONSES TO SPEAKING TASKS 



Listed on the survey response rheet is a series of speaking tasks requiring 
various degrees of language ability to perform. For each task, indicate whether, 
in your professional opinion, bflu^ua) education teachers need to have the 
language ability necessary to cany out the task in order to p erfo rm successfully 
in a bilingual classroom. In other words, for ear* **sk, ask yourself: 



Is the kvd of ability 
required to perform this task 
needed by bilingual education teachers 
in Texas public schools? 



Important The question is NOT Do bilingual teachers need to cany out this 
task in the classroom?" Rather, the question is "Do bilingual education teachers 
need to have the Spanish language ability to cany out this task?" 

Fill in the letter that represents your response to this question in the 
appropriate column on the response sheet The columns are as follows: 

A « Definitely Yes 

B - Probably Yes 

C « Maybe 

D - Probably No 

E - Definitely No 

Following the examples below are detailed descriptions of the speaking tasks. 
Be sure to read them before making your response. 

EXAMPLES 

Here are two example tasks with responses completed for you: 
Example A 

fatenfl an Invitation 

Be able to politely invite someone to your home for a party or other social 
function 

If, in your opinion, bilingual education teachers should dtfinittly have the !■»•! of obility 
required to perform litis speaking task (independent of whether they would need to do the 
task in the classroom), then you would darken circle "A" in the first column of die 
response sheet 

4 
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Example B 



Negotiate Renting Temporary Living Quarters 

Be able to negotiate a rental agreement with a landlord, ask questions about 
what is included in the rent, and ask for clarification of the rental agreement 

If, in your opinion, bilingual education teachers should probably have the iual a! flkilily. 
required to perform this speaking task (independent of whether they would need to do the 
task in the classroom), then you would darken circle m ET in the second column of the 
response sheet 

If you mode the above two responses to die example tasks, your survey response sheet 
would look like this: 



GENERAL PURPOSE DATA SHEET II 
form no. 70921 



Definitely Vo 



Probably >o 



lUyb* 



Probably Tee 



Definitely Tea 



A. Extend an Invitation — — - ■ ■ " 

B. Hcgotlata tenting Teaporary Living Quartcra- 
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Now please make your response for each of the 38 speaking tasks listed on the 
following pages on the appropriate line of the survey response sheet 
Remember to ask yourself, for each task: * 



Is the Iml of ability 
required to perform this task 
needed by bilingual education teachers 
in Texas public schools? 
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SPEAKING TASKS 

Inflate Yowixtf 

Be able to give your name and basic persona! i:\fonnation such as would be 
given at a first meeting. 

2. Explain a Familiar. Simple Process 

Be able to explain how to accomplish everyday processes such as writing e 
check, borrowing a book from the horary, or taking attendance in the classroom. 

3. Describe a Sequence of Events in the Past 

Be able to use and sequence language indicating past time in order to narrate 
an event or incident which occurred recently. 

4. Propose and Defend a Course of Action whh Persuasion 

In light of at least two possible choices of action, be able to propose and defend 
a course of action in such 8 way as to persuade others to accept your choice. 

5. Describe Typical Routings 

Be able to use and sequence language indicating present or habitual time in 
order to narrate recurring events or routines, everyday activities, etc 

& Mf»Ve purchases 

Be able to request hems, discuss prices, and handle currency in a situation 
involving a purchase. 

7. Taft A tovt P erson?) Activities 

Be able to talk about your leisure activities, favorite pastimes, and preferred 
hobbies. 

a H ypothesize Abput an Impersonal Tppic 

Be able to discuss various possibilities (Vhat if situations) surrounding an 
abstract, impersonal topic. 

[FOR SURVEY PURPOSES ONLY] 
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9. Talk Abofl Family Maflbca 

Be able to give the names of the members of your family and simple descriptive 
information, such as their occupations and physical characteristics. 

10. Give a Brief, Organized Factual Summary 

Be able to summarize in an "oral report" fashion factual information about 
topics of 8 personal or professional nature. 

11. State Your Pe rsonal Point of View on a Controversial Subject 

Be able to state what you believe cm a controversial subject and why you hold 
those beliefs. 

12. Describe Expected Future Events 

Be able to use and sequence language indicating future time in order to narrate 
expected occurrences of a personal nature, such as a planned Hp or activity. 

13. Explafn a Com plex Process in Detail 

Be able to explain in detail a non-routine process of an impersonal nature, such 
as how to carry out a scientific investigation or how to write a term paper. 

14. Order a Meal 

Be able to ask questions about menu items, order food, and ask for and settle a 
bill. 

15. Express Personal A poJq p cs 

Be able to apologize dearly and appropriately to an offended party. 

16. G|ve Advice 

Be able to give advice to someone faced with making a decision between two or 
more choices, giving supporting reasons for the advice given. 

17. Hypothesize About a Personal Situation 

Be able to say what you would do in a hypothetical situation. 

[FOR SURVEY PURPOSES ONLY] 
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1& Describe Your Daily Routing 

Be able to narrate your typical daily activities. 

19. Give Instructions 

Be able to give instructions and explain the steps involved in carrying out an 
activity. 

20. Give a Brief Personal History 

Be able to talk about your personal background 

21. State Advantages and Dis advantages 

Be able to state the advantages and disadvantages of a situation (such as living 
in a big city), a decision (such as going to college), or an object that has 
affected society (such as the computer). 

21 Support Opinion* 

Be able to state, support and defend a personaDy-beld opinion or belief about 
an issue. 

23. Describe Health Problems 

Be able to describe health problems or conditions. 

24. Discuss a Professional Topj c 

Be able to discuss at length and in detail a topic of professional interest 

25. Describe a Comply Ohjert j n Prtafl 

Be able to describe a tomplex object such as a car or bicycle in detail and with 
precise vocabulary. 

26. Lodge a Complaint 

Be able to lodge a complaint, giving the reasons for and details behind the 
complaint 

(FOR SURVEY PURPOSES ONLY] 
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27. Talk About Your Future Plans 

Be able to state and describe your personal or professional plans, goals and 
ambitions. 

2& Give a Professional Talk 

Be able to present a talk on a topic of professional interest 

29. Mate Arrangements for Fwtvrc Agnatic 

Be able to inquire about and to make arrangements for future activities, and to 
set the date, time and place. 

30. Evaluate Issues Surroundin g a Conflict 

Be able to present arguments on both sides of a familiar issue or topic and 
evaluate their relative merits. 

31. Give Directions 

Be able to give directions on how to get from one place to another. 

32. Describe a Place 

Be able to describe in detail a particular place, such as a school, a store, or a 
park. 

33. Explain a C omplex Proces s of a Personal Nature 

Be able to describe and explain in detail a non-routine process such as how to 
get a job, or bow to apply to college. 

34. Hypothesize About Probable Outcomes 

Be able to discuss what could happen if something unexpected occurs. 

35. Correct an Unexpected Situation 

Be able to handle an unexpected outcome, such as receiving faulty merchandise. 

[FOR SURVEY PURPOSES ONLY] 
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36. Change Someone'* Behavior through Persuasion 

Be able to persuade someone to do something be or she is not inclined to do, or 
to cease doing something which is annoying to you. 

37. Describe Habitual Actions in the Past 

Be able to describe people, places or things in the past, such as the work 

schedule you used to have or leisure activities you used to da » 

38. Compare and Contrast Two Objects or Places 

Be able to compare and contrast two objects, places, or customs. 



STEP 4 ADDITIONAL COMMENTS 

Please use the space provided in the three WRITE-IN AREAS on the back ot 
the survey response sheet for any additional comments you wish to make 
regarding the oral language functions to be included on the TOPT-Spanish. 



STEP 5 RETURNING THE SURVEY 

Unfold the enclosed pre-addressed, stamped envelope. Insert the blue and white 
machine-readable survey response sheet into the envelope, being careful not to 
fold it Return the machine-readable survey response sheet only as soon as 
possible, but postmarked no later than MAY 4, 1990, to: 

Mr. Dorry Kenyon 
Center for Applied linguistics 
1118 22nd Street, NW 
Washington, DC 20037 



Thank you for your participation in this survey. 



RF1 URN BY MAY 4, 1990 
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Maybe 
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WSf fcO 2 PINCH OMt V ^ " 



r 



Probably Yea 



B — 



Definitely Tea 



A - 



1. 

2. 

3. 

4. 

5. 

6. 

7. 

8. 

9. 

10. 

11. 

12. 

13. 

14. 

15. 

16. 

17. 

18. 

19. 

20. 

21. 

22. 

23. 

24. 

25. 

26. 

27. 

28. 

29. 

30. 

31. 

32. 

33. 

34. 

35. 

36. 

37. 

38. 



Describe Expected Future Events- 
Explain a Complex Process in Detail' 
Order a Meal 
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Introduce Yourself—- — — - --™--— > 

Explain a Familiar, Simple Process ZZZZZZZ> 

Describe a Sequence of Events in the Past > 

Propose * Defend a Course of Action with Persuasion> 

Describe Typical Routines ..... Z 

Make Purchases . ...... Z 

Talk About Personal Activities IZZZZIZZZZ> 

Hypothesize About an Impersonal Topic 

Talk About Family Members * 

Give a Brief, Organized Factual Summary ZZZZ» 

?^<K er 2° nal *2 int of View (Controversial Subject) 
scribe Exoected Ftit-n^ *s,„*~«. 

— — . > 

Express Personal Apologies .ZIZZZZZZZI ^ 

Give Advice ' -> 

Hypothesize About a Personal Situation ZZZZZZZZZZ^ 

Describe Your Daily Routine Z 

Give Instructions " : 

Give a Brief Personal History Z-ZZZZZ 

State Advantag e and Disadvantages 

Support Opinions 

Describe Health Problems ZZZ 

Discuss a Professional Topic ZZZ 

Describe a Complex Object in Detail 

Lodge a Complaint 

Talk About Your Future Plans ZZZ 

Give a Professional Talk - . Z 

Make Arrangements for Future Activities ZZZ'. 

Evaluate Issues Surrounding a Conflict > 

Give Directions Z * 

Describe a Place— — - m > 

H^?i!L?,f 0 ?E le ? * rocess ot * PersonaTNature™^ 
Hypothesize About Probable Outcomes 

Correct an Unexpected Situation— 

Change Someone's Behavior through Persuasion — > 

Describe Habitual Actions in the Past--™- > 

compare and Contrast Two Objects or Places > 

44-6 
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INVITATION TO EXAMINEES TO PARTICIPATE IN TRIALING 

(Example from El Paso) 
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jSS^° Ur re$POn$C * G " iB *" tafonn.i to «..mi mail no later than 

PLEASE PRINT 

Name: , 

S.S.0. Daytime phone; - . 

Address: * Homc Pfcw**: . 



— Y<S ' 1 m * » bte 10 P*^I»te the preliminary administration of the TOPT. 
Language (circle): French Spanish 

Subject: French Language Spanish Language BilinguaJ Education 
I can participate in the following administration of the lest: 



Date: 

Location: 

Time: 



— No. I will not be able to attend. 



* 



Center for Applied Linguistics 
1118 22nd St., NW 
Washington, DC 20037 




Mr. Dorry Kenyon 
Center for Applied Linguistics 
1118 22nd St., NW 
Washington, DC 20037 



us 



~F1~ 
~F3~ 
~F4~ 



Dear ~F2~: 

Your school has identified you as a future Texas foreign language educator and has 
forwarded your name to us. We would like to invite you to participate in the 
development of an oral proficiency test for French, Spanish, and bilingual education 
teachers. 

Beginning in the Spring of 1991, the state of Texas will require prospective teachers of 
French, Spanish, and bilingual education to demonstrate their speaking proficiency via 
the Texas Oral Proficiency Test (TOPT), now under development. You can contribute 
to the development of the TOPT by taking a preliminary form. In doing so, you will: 

-have an opportunity to give feedback about the test, which will be 
considered in writing the final version 

-become familiar with the test you may have to take for certification in 
Texas 

-help ensure that future teachers (perhaps including yourself) will take a 
fair test 

-help ensure high standards for foreign language and bilingual education 
teachers in Texas 

Only 90 minutes of your time is required. During this time, you will take the test and 
complete a feedback questionnaire. 

You may take the test at UT-E1 Paso, Language Lab, Liberal Arts Building, Room 238, 
on Tuesday, June 26, 1990. The test will begin at the following times: 

9:30 a.m. French 
11:00 a.m. Spanish 
1:30 p.m. French 
3:00 p.m. Spanish 

Please indicate whether or not you plan to participate by completing the enclosed post 
card and sending it to the Center for Applied Linguistics. This card must be sent no 
later than June 20, 1990. 
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Remember, the result* of this test administration will not be used to evaluate your 
language proficiency. They will be used only to refine the final version of the test 
Your participation is essential to ensure that the TOPT provides all examinees with an 
opportunity to accurately demonstrate their current ability to speak French or Spanish. 
We hope you will contribute to make the test a success. 

Sincerely, 



Dorry Kenyon 
Project Coordinator 
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YOU ARE INVITED 



to participate in the development of an oral proficiency lest 
for French, Spanish, and bilingual education teachers. 



Beginning in the Spring of 1991, the state of Texas will requiie prospective teachers of 
French, Spanish, and bilingual education to demonstrate their speaking proficiency via 
the Texas Oral Proficiency Test (TOPT), now under development. You can contribute 
to the development of the TOPT by taking a preliminary form. In doing so, you will: 



* have an opportunity to give feedback about the test, which 

will be considered in producing the final version 

* become familiar with the test you may have to 
take for certification in Texas 

* help ensure that future teachers (perhaps including yourself) 

will take a fair test 

* help ensure high standards for foreign language and 
bilingual education teachers in Texas 



Only 90 minutes of your time is required. During this time, you will take the test and 
complete a feedback questionnaire. 

You may take the test at UT-EI Paso, Language Lab, Liberal Arts Building, Room 238, 
on Tuesday, June 26, 1990. The test will begin at the following times: 



If you can participate, please give your name, social security number, and time you plan 
to take the test to your teacher. Your teacher will then give you a post card for you to 
complete and send to the Center for Applied Linguistics. This card must be sent no 
later than June 20, 1990. 

Remember, the results of the test will not be used to evaluate your language 
proficiency. They will be » ltd only to refine the final version of the test. Your 
participation is essential to ensure that the TOPT provides all examinees with an 
opportunity to accurately demonstrate their current ability to speak French or Spanish. 
We hope you wiJJ contribute to this effort. 



9:30 a.m. 
11:00 a.m. 
1:30 p.m. 
3:00 p.m. 



French 
Spanish 
French 
Spanish 



er|c 
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APPENDIX C 

DESCRIPTION OF TOPT TRIALING EXAMINEES 
(By Language and Form) 
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Trial ing: Descriptive Stats/French by form 
- — — TOPT For»»Fore A — 



Trial ing site 





Cuaulativs 


Cumulative 


CITY 


Prsqusncy Percent Frsqusncy 


Percent 


Austin 


3 30.0 3 


30.0 


Arlgton 


3 30.0 6 


60.0 


Hurst 


3 30.0 9 


90.0 


Houston 


1 10.0 10 


100.0 




Currant Status 






Cumulative 


Cumulative 


STATUS 


Frequency Percent Frequency 


Percent 


In-Ssrv 


4 44.4 4 


44.4 


Other 


5 55. o 9 


2.00 . 0 




Frequency Missing - 1 






Certification 






Cumulative 


Cusulative 


CERT 


Frequency Percent Frequency 


Percent 



Sec Fren 4 44.4 4 44.4 

Sec Span 1 11.1 5 55.6 

Fr & Sp 1 11.1 6 66.7 

Other 3 33.3 9 100.0 



Frequency Hissing ■ 1 
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Trial ing: Descriptive 6 tats/ French by For* 
— . — TOFT Fors-Fors A — — — — — - - 



ETHNIC Frequency 



Ethnicity 

emulative Cumulative 
Percent Frequency Percent 



Black 
Other 
White 



1 
2 

7 



10.0 
20*0 
70.0 



1 
3 
10 



10.0 
30.0 
100.0 



Sex 

emulative Cumulative 
SEX Frequency Percent Frequency Percent 



Female 8 80.0 8 80.0 

Kale 2 20.0 10 100.0 



Self Rating 

Cumulative Cumulative 

SELFRATE Frequency Percent Frequency Percent 

High-Sup l ix. 1 1 li.i 

Adv* 3 33.3 4 44#4 

Adv 1 n.l 5 55.6 

Int-H 3 33.3 8 88.9 

Int-VM 1 11.1 9 100.0 

Frequency Missing - 1 



ERIC 
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Trial ing: Descriptive Stats/French by Font 
TOFT Form-Form B — — — — 



Trialing Site 









Cumulative 


Cumulative 




Frequency 


Percent 


Frequency 




Arlgton 


8 


80,0 


8 


80.0 


Hurst 


2 


20.0 


10 


100.0 




Current Status 










Cumulative 


Cumulative 


STATUS 


Frequency 


Percent 


Frequency 


Percent 


Pre-Serv 


3 


30.0 


3 


30.0 


In-Serv 


5 


50.0 


8 


80.0 


other 


2 


20.0 


10 


100.0 




Certification 










Cumulative 


Cumulative 


CERT 


Frequency 


Percent 


Frequency 


Percent 


El Fren 


1 


10.0 


1 


10.0 


Sec Fren 


6 


60.0 


7 


70.0 


Fr 4 Sp 


1 


10.0 


8 


80.0 


Other 


2 


20.0 


10 


100.0 






Ethnicity 










Cumulative 


Cumulative 


ETHNIC 


Frequency 


Percent 


Frequency 


Percent 


Other 


1 


10.0 


1 


10.0 


White 


9 


90.0 


10 


100.0 
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Trial ing: Descriptive Stats/French by Form 

— — — — — TOFT Fon-Fon B - — — — 

Sax 

Cumulative Cumulative 
SEX Frequency Percent Frequency Percent 

Fenale 9 90.0 9 90.0 

Hale 1 10.0 10 100.0 



Self Rating 

Cumulative Cumulative 
SELFRATK Frequency Percent Frequency Percent 



Sup 2 20.0 2 20.0 

Adv+ 3 30.0 5 50.0 

Adv 1 10.0 6 60.0 

Int-H 3 30.0 9 90.0 

Ir,t-L/M 1 10.0 10 100.0 
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Trial ing: Descriptive State/French by For* 
— TOPT Fori-roxi c — — . 



Trial inf Sita 









Cumulative 


Cumulative 


CITY 


Frequency 


Percent 


Frequency 


Percent 


Arlgton 


2 


20.0 


2 


20.0 


Hurst 


5 


50.0 


7 


70.0 


Edinburg 


3 


30.0 


10 


100.0 




Current Status 










Cumulative 


Cumulative 


STATUS 


Frequency 


Percent 


Frequency 


Percent 


Pre-Serv 


2 


20.0 


2 


20.0 


In-serv 


2 


20.0 


4 


40.0 


Other 


6 


60.0 


10 


100.0 




Certification 










Cumulative 


Cumulative 


CERT 


Frequency 


Percent 


Frequency 


Percent 


Sec Fren 


3 


30.0 


3 


30.0 


Fr & Sp 


2 


20.0 


5 


SO.O 


other 


5 


50.0 


10 


100.0 






Ethnicity 










Cumulative 


Cumulative 


ETHNIC 


Frequency 


Percent 


Frequency 


Percent 


Hispan 


2 


20.0 


2 


20.0 


Other 


1 


10.0 


3 


30.0 


White 


7 


70.0 


10 


100.0 
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Trial ing: Descriptive Stats/French by For* 

— — TOFT Fore-Fore C 

Sex 

Cuaulativa Cumulative 
SEX Frequency Percent Frequency Parcent 

Feaale 9 90.0 9 90.0 

Male 1 10.0 10 100.0 



Self Sating 

Cumulative Cumulative 
SELFRATE Frequency Percent . Frequency Percent 



High-Sup 1 10.0 1 10.0 

Sup 1 10.0 2 20.0 

Adv+ 2 20.0 4 40.0 

Adv 1 10.0 5 50.0 

Int-H 4 40.0 9 90.0 

Znt-I/M 1 10.0 10 100.0 
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Trial ing: Descriptive Stats/French by For* 
— — TOPT Form-Form D 



Trial ing site 









Cumulative 


Cumulative 


CITY 


Frequency 


Percent 


Frequency 




Arlgton 


4 


36.4 


4 


36-4 


Hurst 


5 


45.5 


9 




San Ant 


2 




11 


A W e W 




Current Statue 










Cumulative 


Cumulative 


STATUS 


Frequency 


Percent 


Frequency 




Pre-Serv 


1 


9.1 


1 


9.1 


In—Serv 


4 


36.4 


5 


e!9 % 


Other 


6 


54.5 


11 


AW • w 




Certification 










Cumulative 


Cumulative 


CERT 


Frequency 


Percent 


Frequency 


Percent 


Sec Fren 


3 


27.3 


3 


27.3 


Sec Span 


1 


9.1 


4 


36.4 


Fr ft Sp 


3 


27.3 


7 


63 6 


Other 


4 


36.4 


11 


inn n 






Ethnicity 










Cumulative 


Cumulative 


ETHNIC 


Frequency 


Percent 


Frequency 


Percent 


Hi span 


1 


9.1 


1 


9.1 


Other 


1 


9.1 


2 


18.2 


White 


9 


81.8 


11 


100.0 
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Trilling: Descriptive Stats/FJencn by Form 
........... ™..— TOPT For»»For» D — — — — - 

Sex 

Cunulative Cueuletive 
SEX Frequency Percent Frequency Percent 

Feaale 8 72. V 0 72.7 

Male 3 27.3 XI 100.0 



Self Rating 

Cunulative Cumulative 

SELFRATE Frequency Percent Frequency Percent 

Sup 3 27.3 3 27.3 

Adv* 2 18.2 5 45.5 

Adv 2 18.2 7 63.6 

Int-H 4 36.4 11 100.0 
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Trial ing: Descriptive Stats/TOPT-Spanish by Form 
— — TOPT Fore-Form A — 



Trial ing Site 







emulative 


Cumulative 


CITY 

W AAA 


Frequency 


Percent Frequency 


Percent 


El Paso 


3 


11.5 3 


11.5 


Austin 


8 


30.8 11 


42.3 


Hurst 


2 


7.7 13 


50.0 


Edinburg 


11 


42.3 24 


M± A rife 

92. 3 


Houston 


2 


7.7 26 


100.0 




Current Status 








Cumulative 


Cumulative 


STATUS 


Frequency 


Percent Frequency 


Percent 


Pre-Serv 


14 


56.0 14 


56.0 


Jn-Serv 


5 


20.0 19 


76.0 


Other 


6 


24.0 25 


100.0 




Frequency Missing « 1 






Certification 








Cumulative 


Cumulative 


CERT 


Frequency 


Percent Frequency 


Percent 


El Span 


2 


8.0 2 


8.0 


Sec Span 


7 


28.0 9 


36.0 


Bil Ed 


13 


52.0 22 


88.0 


Other 


3 


12.0 25 


100.0 



Frequency Hissing ■ 1 
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Trial ing: Descriptive Stata/TOFT-Spanieh by Fori 

TOPT Form«Pom A 

Ethnicity 







fiarrAnf 
rRi wvnv 


Cumulative 


Cumulative 

eT V A VVlIb 


Hispan 


19 


73.1 


19 


73.1 


White 


7 


26.9 


26 


100.0 






Sex 












Cumulative 


Cumulative 


SEX 


Frequency 


Percent 


Frequency 


Percent 


Female 
Male 


18 
8 


69.2 
30.8 


18 

26 


69.2 
100.0 



Self Rating 

Cumulative Cumulative 

SELFRATE Frequency Percent Frequency Percent 

High-Sup 1 3.8 1 3.8 

Sup 2 7.7 3 11.5 

Adv+ 14 53.8 17 65.4 

Adv 5 19.2 22 84.6 

Xnt-H 3 11.5 25 96.2 

Int-L/M 1 3.8 26 100.0 
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Trial ing: Descriptive Stats/TOPT-Spanish by Form 












Trlaling Slta 








Cumulative 


Cumulative 


CITY 


Frequency 


Percent Frequency 


Percent 


El Paso 


3 


10.7 3 


10.7 


Austin 


10 


35.7 13 


46.4 


Hurst 


2 


7.1 15 


53.6 


Edinburg 


IX 


39.3 26 


92.9 


San Ant 


X 


3.6 27 


96.4 


Houston 


1 


3.6 28 


100.0 




Current Statua 








emulative 


Cumulative 


STATUS 


Frequency 


Percent Frequency 


Percent 


Pre-Serv 


16 


64.0 16 


64.0 


In-Serv 


4 


16.0 20 


80.0 


Other 


5 


20.0 25 


100.0 




Frequency Missing * 3 






Certification 








Cumulative 


Cumulative 


CERT 


Frequency 


Percent Frequency 


Percent 


El Span 


3 


11.5 3 


11.5 


Sec Span 


9 


34.6 12 


46.2 


Bil Ed 


11 


42.3 23 


88.5 


Other 


3 


11.5 26 


100.0 




Frequency Hissing * 2 
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Trial ing: Descriptive Stat»/TOPT-Spanish by For* 















Ethnicity 










Cumulative 


Cumulative 


ETHNIC 


Frequency 


ffw* vVIl W 


Preouencv 


Percent 


hi span 


22 

e» m 


78.6 


22 


7B 6 


White 


6 


21.4 


28 


100.0 






Sex 












Cumulative 


Cumulative 


SEX 


Frequency 


Percent 


Frequency 


Percent 


Female 


20 


71.4 


20 


71.4 


Kale 


8 


28.6 


28 


100.0 



Self Rating 

Cumulative Cumulative 

SELFRATE Frequency Percent Frequency Percent 

High-Sup 4 16.0 4 16.0 

Sup 6 24.0 10 40.0 

Adv+ 8 32.0 18 72.0 

Adv 2 8.0 20 80.0 

Int-H 3 12.0 23 92.0 

Int-VM 2 25 100.0 

Frequency Missing * 3 
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Trial ing: Descriptive Stats/TOPT-Spanish by Form 
TOPT Form-Form C 



Trial ing Sits 









Cumulative 


Cumulative 


CITY 


Frequency 


Percent 


Frequency 


Percent 


El Paso 


2 


5.3 


2 


5.3 


Austin 


3 


7.9 


5 


13.2 


nuriv 


1 


2.« 


6 


15.8 


Edinburg 


15 


39.5 


21 


55.3 


San Ant 


14 


36.8 


35 


92.1 


Houston 


3 


7.9 


38 


100.0 




Current Status 










Cumulative 


Cumulative 


STATUS 


Frequency 


Percent 


Frequency 


Percent 


Pre-Serv 


28 


73.7 


28 


73.7 


In-Serv 


4 


10.5 


32 


84.2 


Other 


6 


15.8 


38 


100.0 




Certification 










Cumulative 


Cumulative 


CERT 


Frequency 


Percent 


Frequency 


Percent 


El Span 


4 


10.5 


4 


10.5 


Sec Span 


5 


13.2 


9 


23.7 


Bil Ed 


26 


68.4 


35 


92.1 


Other 


3 


7.9 


38 


100.0 
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Trial ing: Descriptive Stat»/T0PT-5pani»h by Font 

— ' TOPT Fore-Fore C — — — - — 

Ethnicity 



ETHNIC 


Frequency 


Percent 


Cumulative 
Frequency 


emulative 
Percent 


Black 


1 


2.6 


1 


2.6 


Hispan 


30 


78.9 


31 


81.6 


White 


7 


18.4 


38 


100.0 






Sex 












Cumulative 


Cumulative 


SL'X 


Frequency 


Percent 


Frequency 


Percent 


Feaale 
Male 


32 
6 


84.2 
15.8 


32 
38 


84.2 
100.0 



Self Rating 

Cumulative Cumulative 

SELFRATE Frequency Percent Frequency Percent 

High-Sup 6 16.2 6 16.2 

Sup 7 18.9 13 35.1 

Adv+ 10 27.0 23 62.2 

Adv 8 21.6 31 83.8 

Int-H 5 13.5 36 97.3 

Int-L/M 1 2.7 37 100. 0 

Frequency Missing * 1 
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Trial ing: Descriptive Stats/TOPT-Spanish by Form 
.... TOPT For»-for» D — — — — 



Tripling Site 

Cumulative 

CITY Frequency Percent Frequency 



Cumulative 
Percent 



VI fiaan 
XtX raio 


j ii . i <» 


AA * A 


Austin 


4 14.8 7 


25.9 


Hurst 


4 14.8 11 


40.7 


Edinburg 


16 59.3 27 


100.0 




Current Status 






^ti mil 1 4 vi 

VMJBU«| OV1V1 


Oil win 1 a ,va 

VUIBMi. QV1V6 


STATUS 


Frequency Percent Frequency 


Percent 


•IB vCl v 


14 53 8 11 


53.8 

mm* ajP • SP 


In^Strv 

4S> # 1 ^7 V ak V 


W e# • 1 • W 


e w <e 7 




£ 23 1 2fi 

V a e «s> (S» e# 


avv • v 




Frequency Missing ■ 1 






Certification 






Cumulative 


Cumulative 


CERT 


Frequency Percent Frequency 


Percent 


El Span 


2 7.7 2 


7.7 


Sec Span 


5 19.2 7 


26.9 


Bil Ed 


16 61.5 23 


88.5 


Other 


3 11.5 26 


100.0 



Frequency Missing » 1 



0 
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Trial ing: Descriptive Stats/TOPT-Spanlsh by Ton 
— — — — TOPT Fore-Fore D : — — - 



Ethnicity 



ETHNIC 


Frequency 


Percent 


Cumulative 
Frequency 

jap asv "^bp- ^am ^» bp m ^aa^ 


Cumulative 
Psrcsm. 


Hi span 


24 


88.9 


24 


88.9 


Whits 




11.1 


27 


100.0 






Sex 












Cumulative 


Cumulative 


SEX 


Frequency 


Percent 


Frequency 


Percent 


Fenale 
Mais 


18 
9 


€6.7 
33.3 


18 
27 


66.7 
100.0 



Self Rating 

Cui .ive Cumulative 



SELFRATE 


Frequency 


Percent 


Fr&^wency 


Percent 


High-Sup 


2 


8.0 


2 


8.0 


Sup 


5 


20.0 


7 


28.0 


Adv+ 


8 


32.0 


15 


60.0 


Adv 


3 


12.0 


18 


72.0 


Int-H 


4 


16.0 


22 


88.0 


Int-L/M 


2 


8.0 


24 


96.0 


Novice 


1 


4.0 


25 


100.0 



Frequency Missing ■ 2 
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APPENDIX D 



TOPT TRIALING EXAMINEE RESPONSE FORM (Part I): 
QUANTITATIVE DATA COLLECTION INSTRUMENT FOR 

BACKGROUND DEMOGRAPHIC INFORMATION 
AND ATTITUDES TOWARDS THE TEST AND ITS ITEMS 
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Texas Oral Proficiency Test (TOPT) 
Trialing Feedback 

Please do not write in this booklet! 

INSTRUCTIONS 

Thank you very much for participating in this trialing of the TOPT. Your comments on 
the test are valued and will be given full consideration in the test revision process 
before the TOPT is finalized. Your frank input on the test will help ensure that the 
final version of the TOPT is a fair test that allows all examinees to demonstrate their 
current ability to speak French or Spanish. 

Your feedback on the test is being collected in two formats. The first (Part I) is 
through your responses to statements in this booklet. You will record your responses 
on the blue, machine-readable response sheet The second (Part II) allows you to write 
your own comments in response to the issues raised in Part I. It also allows you to 
describe any concerns you have about the test. You will record your responses to this 
part in the white booklet 

IMPORTANT: In giving your feedback on the test, please remember that the purpose 
of the TOPT is to provide each candidate with the opportunity to demonstrate his or 
her current ability to speak French or Spanish. In other words, its purpose is to 
capture on tape a "snapshot" of one's ability to speak French or Spanish. Think about 
your own performance on the TOPT you just completed. Our goal in developing the 
TOPT is to ensure that a person listening to the tape containing your responses gets an 
accurate picture of your current ability to speak French or Spanish. 

PART I. MACHINE-READABLE RESPONSE SHEET. (Use a No. 2 pencil to mark 
your responses.) 

STEP 1 IDENTIFICATION 

Please fill out the information requested in the upper right-hand corner of the 
blue machine-readable response sheet. Be sure to circle the language in which 
you took the TOPT (French or Spanish), the form of the TOPT you took (A, B, 
C, or D), and the subject area you are preparing to teach or are already teaching 
(French, Spanish, or bilingual education). 

STEP 2 ID Number 

Please write your social security number in the boxes in the area entitled ID 
NUMBER on the top left-hand corner of the machine-readable response sheet 
Then fill in the circle corresponding to the number in each box. 



1 



o 
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Please do not write in this booklet! 



STEP 3 BACKGROUND INFORMATION 

For demographic purposes, please answer each lettered question presented on 
the next page in the box labeled BACKGROUND INFORMATION. Write your 
answer in the area entitled SPECIAL CODES on the top left-hand corner of the 
response sheet. For each lettered question (A through G), write the number of 
your answer in the block on the answer sheet. Then fill in the circle 
corresponding to the number of your answer. 
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BACKGROUND INFORMATION 

A. For which language did you take the TOPT? 
(0) French (1) Spanish 

B. Which form of the TOFT did you take? 



(0) Form A 

(1) Form B 



(2) FormC 

(3) FormD 



C In which city did you take the TOPT? 



(0) ElPaso 

(1) Austin 

(2) Arlington 

(3) Hurst 



(4) Edinburg 

(5) San Antonio 

(6) Houston 



D. What is your current status in respect to teaching? 

(0) Pre-service (not yet certified) 

(1) In-service (certified in Texas and teaching in the classroom) 

(2) Other 

E. Which type of certification do you have or will you be seeking? 

(0) Elementary certificate with French specialization 

(1) Secondary French certificate 

(2) Elementary certificate with Spanish specialization 

(3) Secondary Spanish certificate 

(4) Certificate or endorsement in bilingual education 

(5) (l)and(3) 

(6) Other 



F. What is your ethnic group? 

(0) Hispanic 

(1) Black 

G. What is your sex? 
(0) Male 



(2) White 

(3) Other 



(1) Female 



3 
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Please do not write in this booklet! 



STEP 4 SELF RATING 

We would like you to describe your ability to speak French or Spanish. Below 
are seven descriptions of different levels of ability, ordered from high to low. In 
the box labeled "J" in the area entitled SPECIAL CODES on the machine- 
readable response sheet, please write in the number of the description below that 
most accurately represents your ability to speak French or Spanish. After you 
have written in your answer, fill in the circle corresponding to the number of 
your answer. 

(0) I can speak the language about as well as a well-educated native speaker and 
can handle sophisticated language tasks such as public speaking, formal 
interpreting, etc. 

(1) Using a standard or international form of the language, I can participate 
effectively and with ease in both formal and informal conversations on abstract 
and professional topics, as well as on practical and social topics. I can di>cuss 
my particular interests and fields of competence with ease. 

(2) I can handle a broad variety of everyday, school, and work situations relating to 
my particular interests and fields of competence. I am usually, though not 
always, effective in supporting my opinions and explaining or describing things in 
detail. 

(3) I can handle informal conversations successfully. That is, I can begin, continue, 
and bring to completion a wide variety of conversational tasks, including those 
involving a complication or those generated by an unforseen turn of events. 
Using general vocabulary, I can communicate facts and talk casually about topics 
of current public interest and of personal interest. 

(4) I can handle most uncomplicated communication tasks and social situations. For 
example, I can discuss my background, interests, and leisure time activities. I 
have some ability, although limited, to converse on impersonal topics such as 
current events. 

(5) I can handle very simple face-to-face conversations on familiar topics such as my 
family, the weather, food, clothing, etc. I can ask and answer simple questions, 
usually without difficulty. 

(6) My ability to ask and answer questions is limited to the use of memorized 
utterances, although I occasionally speak in sentences. 



4 
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STEP 5 RESPONSES TO STATEMENTS ABOUT THE TOPT 

Listed below (and abbreviated on the blue machine-readable response sheet) are 
a number of statements about the individual items on the TOPT and on the test 
in general. For each statement, fill in the letter under the column that best 
reflects the degree to which you agree with the statement The columns are as 
follows: 

A « Strongly Agree 

B b Agree 

C ■ Agree and Disagree 

D ■» Disagree 

E ■ Strongly Disagree 

Please feel free to use your test booklet to refresh your memory about the items 
as you respond to the statements. 

Statements about indiv idual itemfr , 

1. The opening conversation with the native speaker helped ease me into the 
testing situation. 

2. In the opening conversation, the time allowed for making my responses was in 
general, appropriate. r 

3. As a whole I felt picture item #1 allowed me to give a response that reflects my 
current ability to speak the language. 

4. The map for picture item #1 was clear and understandable. 

5. In picture item #1, the time allowed for preparing my answer and making my 
response was appropriate. 

6. As a whole, I felt picture item #2 allowed me to give a response that reflects my 
current ability to speak the language. 

7. The drawing for picture item #2 was clear and understandable. 

8. In picture item #2, the time allowed for preparing my answer and making my 
response was appropriate. 

9. As a whole, I felt picture item #3 allowed me to give a response that reflects my 
current ability to speak the language. 
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10. The pictures for picture item #3 were clear and understandable. 

11. In picture item #3, the lime allowed for preparing my answer and making my 
response was appropriate. 

12. As a whole, I felt picture item 04 allowed me to give a response that reflects my 
current ability to speak the language. 

13. The pictures for picture item 04 were clear and understandable. 

14. In picture item #4, the time allowed for preparing my answer and making my 
response was appropriate. 

15. As a whole, I felt picture item 05 allowed me to give a response that reflects my 
current ability to speak the language. 

16. The pictures for picture item #5 were clear and understandable. 

17. In picture item #5, the time allowed for preparing my answer and making my 
response was appropriate. 

18. I felt topic item #1 allowed me to give a response that reflects my current ability 
to speak the language. 

19. In topic item #1, the time allowed for preparing my answer and making my 
response was appropriate. 

20. I felt topic item 02 allowed me to give a response that reflects my current ability 
to speak the language. 

21. In topic item 02, the time allowed for preparing my answer and making my 
response was appropriate. 

22. I felt topic item #3 allowed me to give a response that reflects my current ability 
to speak the language. 

23. In topic item #3, the time allowed for preparing my answer and making my 
response was appropriate. 

24. I felt topic item 04 allowed me to give a response that reflects my current ability 
to speak the language. 

25. In topic item #4, the time allowed for preparing my answer and making my 
response was appropriate. 
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26. I felt topic item #5 allowed me tc give a response that reflects my current ability 
to speak the language. 

27. In topic item #5, the time allowed for preparing my answer and making my 
response was appropriate. 

28. I felt situation Item #1 allowed me to give a response that reflects my current 
ability to speak the language. 

29. In situation item #1, the time allowed for preparing my answer and making my 
response was appropriate. 

30. I felt situation item #2 allowed me to give a response that reflects my current 
ability to speak the language. 

31. In situation item #2, the time allowed for preparing my answer and making my 
response was appropriate. 

32. I felt situation item #3 allowed me to give a response that reflects my current 
ability to speak the language. 

33. In situation item #3, the time allowed for preparing my answer and making my 
response was appropriate. 

34. 1 felt situation item #4 allowed me to give a response that reflects my current 
ability to speak the language. 

35. In situation item #4, the time allowed for preparing my answer and making my 
response was appropriate. 

36. I felt situation item #5 allowed me to give a response that reflects my current 
ability to speak the language. 

37. In situation item #5, the time allowed for preparing my answer and making my 
response was appropriate. 
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Statements ab out the test in general. 

38. I was not unduly nervous during the test. 

39. I would prefer that a "beep" signal be used in place of the French or Spanish 
speaker as a signal to begin my response after preparing my answer. (In other 
words, I would prefer that the only place French or Spanish be heard is in the 
opening conversation.) 

40. A person listening to the tape containing my responses will get an accurate 
picture of my current ability to speak French or Spanish. 



PART II. ADDITIONAL COMMENTS f White Booklet^ 

The white booklet contains spaces for your comments on the individual TOPT 
items and on the test in general. 

STEP 1 IDENTIFICATION 

Please fill out the information requested at the top of i\e first page. 

STEP 2 COMMENTS 

The white booklet gives you three sections in which to write your comments on 
the TOPT. Directions are given in the booklet. 

STEP 3 RETURN ALL MATERIALS TO THE TEST ADMINISTRATOR 

Before leaving the examination room, you will need to return 



1. 


your response tape, 


2. 


the test booklet, 


3. 


this instruction booklet, 


4. 


the machine-readable response sheet, 


5. 


the white response booklet, and 


6. 


your pencil (if borrowed). 



Thank you for your participation in this dialing of the TOPT! 
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GENERAL PURPOSE DATA SHEET II 
form no. 7092 1 



Strongly Disagree 
Disagree 
f Agree end Disagree 
Agree 



use no j pcscu oslv 



Strongly Agree 



1. The opening conversation helped ease me... — > 

2. Opening conversation time allowed was appropriate — > 

3. Picture item #1 allowed a response that reflects—- > 

4. The map for picture item #1 was clear > 

5. Picture item fl tine allowed for preparing answer— > 

6. Item 42 allowed response that reflects my ability— > 

7. The drawing for picture item 42 was clear -> 

8. Picture item 42 time allowed for preparing answer— > 

9. Picture item 13 allowed a response that reflects — -> 

10. The pictures for picture item #3 were clear -> 

11. Picture item #3 time allowed for preparing answer — > 

12. Picture item 44 allowed a response that reflects — -> 

13. The pictures for picture item #4 were clear > 

14. Picture item 14 time allowed for preparing answer— > 

15. Picture item #5 a response that reflects ability > 

16. The pictures for picture item IS were clear -> \ 

17. Picture item #5 time allowed for preparing answer— > 

18. Topic item #1 allowed a response that reflects — >| 

19. Topic item #1 time allowed for preparing answer > 

20. Topic item 12 allowed a response that reflects > 

21. Topic item #2 time allowed for preparing answer — — > 

22. Topic item #3 allowed a response that reflects -> 

23. Topic item #3 time allowed for preparing answer > 

24. Topic item 44 allowed a response that reflects -> 

25. Topic item 44 time allowed for preparing answer -> 

26. Topic item #5 allowed a response that reflects -> 

27. Topic item 45 time allowed for preparing my answer-> 

28. Situation item II allowed a response that reflects-^ 

29. Situation item 41 time allowed for preparing answer> 

30. Situation item 42 allowed a response that reflecta->| 

31. Situation item 42 time allowed for preparing answer> 

32. Situation item 43 allowed a response that ref lects-> [ 

33. Situation item 43 time allowed for preparing answer> 

34. Situation item 44 allowed a response that reflects-> 

35. Situation item 44 allowed for preparing my answer— > 

36. Situation item 45 allowed a response that reflects-* 

37. Situation item 45 time allowed for preparing my— — > 

38. I was not unduly nervous during the test- — — > 

39. I would prefer a beep signal as a signal to begin— > 

40. A person listening will get an accurate picture—— > 
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TOPT TRIALING EXAMINEE RESPONSE FORM (Part II): 
QUALITATIVE DATA COLLECTION INSTRUMENT FOR 
EXAMINEE COMMENTS ON THE 
TEST AND ITS ITEMS 
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TOPT Trialing Feedback Form 
Part II 



NAME 

SOCIAL SECURITY NUMBER 



DATE 



Please circle your answers 
TOPT Language: FRENCH 
TOPT Form: A B 
Your Teaching Area: French 
Test Site: El Paso 

Edinburg 



SPANISH 

C D 

Spanish bilingual education 

Austin Arlington Hurst 



San Antonio Houston 

Thank you very much for participating in this trialing of the TOPT. Your comments on the test are 
valued and will be given full consideration during the revision process before the final version is printed. 
Therefore, please answer the questions below as clearly and as frankly as you can. 

Part A. In the outline below, comment on any Items to which you awarded a C, D or E in Part L 
These may be items you felt were unclear, unfair or otherwise problematical for you. Especially 
comment on anything that you feel interfered with your ability to answer the question to the best of your 
present ability to speak French or Spanish. Such things might be unclear directions, unclear pictures, 
unrealistic situations, too little time, etc.. Feel free to suggest revisions. 
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Item 

"Opening Conversation" 


Comments 


Picture item #1 

t 

i 




Picture item #2 




Picture item #3 




150 



Picture item I 4 




Picture item #5 





Topic 11 




„ Topic f 2 




Topic 13 




Topic 14 




Topic 15 
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Situation il 



Situation #2 



Situation #3 



"Situation #4 



Situation #5 



Part B. Use the spaces below to make any comments on the statements about the test in general. 

1. I was not unduly nervous during the test. 

Comments? 



152 



2. I would prefer that a "beep" signal be used in place of the French or Spanish speaker as a signal to 
begin my response after preparing my answer. (In other words, I would prefer that the only place 
French or Spanish be heard is in the opening conversation.) 

Comments? 



3. A person listening to the tape containing my responses will get an accurate picture of my current 
ability to speak French or Spanish. 

Comments? 



Part C. Please use the rest of this page to comment on any aspect of the test that is not covered in any 
of the preceding questions. We would especially appreciate any suggestions as to how this test might 
be improved. Thank you very much! 
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EVALUATION SHEET AND INSTRUCTIONS 
USED BY TOPT TRIALING JUDGES AND OBSERVERS 
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TOPT Trialing 
Observer's Evaluation Sheet 

Instructions 

INTRODUCTION. The TOPT is intended to elicit from each examinee a speech sample suitable 
for rating on the ACTFL scale. Alternatively phrased, the goal of the TOPT is to provide each 
candidate with the opportunity to demonstrate his or her current ability to speak French or 
Spanish. The tape of recorded responses should present a "snapshot" of the individual's ability 
to speak French or Spanish and should convey an accurate picture of the candidate's strengths 
and weaknesses. 

First, before you begin listening to the candidate, till out all information requested at the top of 
the form (except for the examinee level estimation). 

Remember that the two purposes for observing the candidate's performance are 1) to judge 
whether items are doing their job of allowing candidates to show what they can do and 2) to 
inform the test revision process. Thus, you should make recommendations about how to improve 
the test and its items if they are not functioning as intended. There are four main areas on which 
you need to comment. 

1. TIME It is important that candidates have an appropriate amount of time for their responses. 
The majority of candidates should have time to give a complete response without having to wait 
during a long silence for the next item to begin. Waiting can create nervousness. On the other 
hand, if candidates are interrupted too often by the next question, they can also get nervous. If 
a candidate is cut off because time is too short, there should be enough of a sample of the type 
of speech elicited by that item on the tape to give the rater a good idea of the c "^ date's ability 
to deal with that language function and of where the candidate would have gone u at or she had 
had more time. 

On the observer response sheet, your feedback on time problems is requested in the area for each 
item marked T". 

No Problem with Time Circle "NP" if there was there was No Problem with time for this 

candidate on that item. (This includes being cut off but still giving 
an appropriate sample.) 

Too Much Time Allowed If too much time was a problem, mark the timeline with an "X" to 

show the approximate number of seconds the candidate had to wait 
for the next item to begin (under the + + + area). Example for 
7-8 seconds too much time: 

+ + + 

15~10)(5*-0--5~10~15 




Too Little Time Allowed If too little time was a ; bltm, mark the approximate number of 

additional seconds the candidate could have used in order to 
demonstrate his or her ability with this task (under the-*- area). 
Example for 5 seconds too litle time: 

+ + + . • . 

15-10-5-0-X-1O-15 



Note: Marks in between the five second intervals printed are allowed and encouraged; i.e, a mark 
between a "5" and a "0" indicates that 2-3 seconds are intended. 

2. CONTRIBUTION OF RESPONSE TOWARDS MAKING AN ACTFL RATING. It is 

important to know something about the quality of the speech sample elicited from the candidate 
by the item. For each picture, situation, and topic item, and for the opening conversation as a 
whole, you are asked to make a judgment on the quality of the speech elicited in terms of the 
goal of getting a ratable speech sample. Make your judgment in the area marked "C. Circle 
either a 1, 2, or 3 to show your judgement, where: 

1 « Speech elicited by item not useful in making a rating 

2 ■ Speech elicited by item useful in making a rating 

3 * Speech elicited by item very useful in making a rating 



3. COMMENTS. The rest of the area is for your comments. You should consider the following: 

(a) BLOCKING: Comment on anything that appeared to block the examinee's response; 
i.e., did not allow the examinee to give as complete a response as he or she may have 
been able. It could be unusual %-jcabulary items, in which case write the offending 
word. It could also be an unclear understanding of the directions to the item. It could 
also be a question of time, i.e., not enough time to think about an answer. It could also 
be a problem with the French or Spanish prompt following the English task description. 

(b) OTHER PROBLEMS: Comment on other problems you notice with the item on the 
basis of the candidate's performance on it. 

(c) RECOMMENDATIONS: Be sure to write down any ideas that come to mind to 
remedy problems you have noticed during observation. 

NOTE: If you are observing the candidate live, be sure to write down any questions you may 
have for him or her about any unexpected performance or behavior you observe durinn the takinc 
of the test. * * 

4. EXAMINEE LEVEL ESTIMATION. After you listen to the candidate, please estimate, to the 
best of your ability, the examinee's ACTFL level. Write this estimate in the space provided on 
the top of the first page of the form. This is NOT intended as an official rating. Your estimation 
will solely be used to help select tapes for rating in the nexi phase of this project 
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TOPTTrialing 
Observer's Evaluation Sheet 

Name of Examinee 

Social Security Number 

TOPT Language: FRENCH SPANISH 

TOPT Form: A B C D 

Examinee's Teaching Area: French Spanish bilingual education 

Observer's Name 

Date of Observation 

Examinee Level Estimation 

Conversation 
1. 

T: NP 

+ < + 

15-10-5-0-5-10-15 



1 

T: NP 

j | j m m m 

15-10-5-0-5-10-15 



3. 

T: NP 

+ + + 

15-10-5-0-5-10-15 



4. 

T; NP 

+ + + 

15-10-5-0-5-10-15 



5. 

T: NP 

+ + + 

15-10-5-0-5-10-15 
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6. 

T: NP 
+ + + 

15-10-5-0-5-10-15 



7. 

T: NP 

+ + + 

15-10-5-0-5-10-15 



8. 

T: NP 

+ + + 

15-10-5-O-5-10-15 



9. 

T: NP 

+ + + 

15-10-5-0-5-10-15 



10. 

T: NP 

+ + + 

15-10-5-0-5-10-15 

(for total opening conversation) 
C: 1 2 3 



Picture #1 
T: NP 

+ + + 

15-10-5-0-5-10-15 

C: 1 2 3 



Picture #2 
T: NP 

+ + + 

15-10-5-0-5-10-15 



C: 1 2 3 



I 



Picture #3 
T: NP 

+ + + - - - 

15-10-5-0-5-10-15 

C: 1 2 3 



Picture #4 
T: NP 

+ + + 

15-10-5-0-5-10-15 

C 123 



Picture #5 
T; NP 

+ + + 

15-10-5-0-5-10-15 

C 1 23 



Topic #1 
T: NP 

15-10-5-0-5-10-15 
C: 1 2 3 



Topic #2 
T: fir 

+ + + 

15-10-5-0-5-10-15 
C: 1 2 3 



Topic #3 
T: NP 

+ + + 

15-10-5-0-5-10-15 

C; 1 23 



Topic #4 
T: NP 

+ + + 

15-10-5-0-5-10-15 

C: 1 2 3 
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Topic #5 
T: NP 

+ + + 

15-10-5-0-5-10-15 

G 1 2 3 



Situation #1 
T: NP 

+ + + 

15-10-5-0-5-10-15 

C: 1 2 3 



Situation #2 
T: NP 

+ + + 

15-10-5-0-5-10-15 

G123 



Situation #3 
T: NP 

+ + + 

15-10-5-0-5-10-15 

C: 1 2 3 



Situation #4 
T: NP 

+ + + 

15-10-5-0-5-10-15 

C: 1 2 3 



Situation #5 
T: NP 

+ + + 

15-10-5-0-5-10-15 

C: 1 2 3 

PLEASE MAKE ANY ADDITIONAL COMMENTS HERE: 
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MEAN RATINGS OF TOPT ITEMS BY TRIALING EXAMINEES 

(By Language and Form) 



1 61 



Trial ing: Descriptive Stats/French by For* 
— — -™- TOPT Pon-Fon A — 



II Obe Variable Label 



Mean 



10 



XTEM1 

ITEM2 

ITEM3 

I TEH 4 

ITEM 5 

ZTEM6 

XTEN7 

ZTEM8 

ITEM9 

ITEM10 

ITEM11 

ZTEM12 

ZTEM13 

ZTEM14 

ZTEM15 

ZTEM16 

ZTEM17 

ZTEM18 

ZTEM19 

ITEM20 

ZTEM21 

ZTEM22 

1TEM23 

ITEM24 

ZTEM25 

ZTEM26 

ZTEM27 

ZTEM28 

ZTEM29 

ZTEM30 

ZTEM31 

ZTEM32 

ZTEM33 

ITEM34 

ITEM35 

ITEM36 

ITEM37 

ZTEM38 

ITEM39 

ITEM40 



Opening Conversation—Central 
Opening Conversation— Tiae 
Picture #1— General 
Picture 41— Map 
Picture #1— Tiae 
Picture #2 — General 
Picture 12 — Picture 
Picture 12— Tiae 
Picture |3 — General 
Picture #3— Pictures 
Picture 13— Tiae 
Picture #4— General 
Picture #4 — Pictures 
Picture #4— Tiae 
Picture 15 — General 
Picture 15 — Pictures 
Picture #5— Tiae 
Topic #1 — General 
Topic #1— Tiae 
Topic 12— General 
Topic 12— Tiae 
Topic #3 — General 
Topic #3— Tine 
Topic #4 — General 
Topic #4— Tiae 
Topic #5 — General 
Topic #5— Tiae 
Situation #1— General 
Situation #1— Tiae 
Situation #2 — General 
Situation 12 — Tiae 
Situation |3 — General 
Situation #3 — Tiae 
Situation #4 — General 
Situation #4— Tine 
Situation #5 — General 
Situation #5— Tine 
Unduly Nervous? 

Replace Target Language Pronpt? 
An Accurate Picture? 



9 
9 



4.111 

4.111 

3.750* 

4.000 

4.111 

4.250 

4.250 

3.889 

4.000 

4.556 

3.778 

4.333 

4.111 

4.333 

4.222 

3.625* 

4.111 

4.444 

4.111 

4.111 

3.889 

4.222 

4.111 

4.111 

4.111 

4.333 

4.111 

4.000 

4.222 

3.667* 

4.333 

4.000 

4.222 

4.000 

4.222 

3.889 

3.556* 

4.222 

2.222 

3.667 



1B2 
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Trial ing: Descriptive State/French by For» 
TOPT Pore-Fore B 



N Obs Variable Label 



10 XTEMl Opening Conversation— General 

ITEM2 Opening Conversation— Tiae 

ITEM3 Picture #1— General 

ITEM4 Picture #1— Map 

ITEMS Picture #1— Ti»e 

ITEM6 Picture #2 — General 

XTEM7 Picture #2— Picture 

XTEM8 Picture 12 — Time 

ITSM9 Picture #3— General 

XTEM10 Picture 13— Pictures 

ZTEM11 Picture I 3 — Ti»e 

ITEM12 Picture #4— General 

ITEM13 Picture #4— Pictures 

2TEM14 Picture #4— Ti«e 

ITEM 15 Picture 15— General 

ITEM16 Picture 15— Pictures 

ZTEM17 Picture 1 5— Tiee 

2TEM18 Topic #1 — General 

ITEM19 Topic #1— Ti»e 

ITEM20 Topic #2— General 

ITEM21 Topic #2— Ti»e 

ITEM22 Topic #3— General 

ITEM23 Topic #3— Ti»e 

I TEH 2 4 Topic #4— General 

ITEM25 Topic #4— Ti»e 

ITEM26 Topic #5— General 

XTEM27 Topic #5 — Tiee 

ITEM28 Situation #1 — General 

ITEM29 Situation #1 — Tiee 

ITEM30 Situation #2 — General 

ITEM31 situation #2— Time 

ZTEM32 Situation #3 — General 

ITEM33 Situation 13— Tiee 

ITEM34 Situation 14— General 

ITEM35 Situation f 4— Time 

ITEM36 Situation #5 — General 

ITEM37 Situation 15— Tiee 

ITEM38 Unduly Nervous? 

ITEM39 Replace Target Language Prompt? 

ITEM40 An Accurate Picture? 



M 


Mean 


10 


3.000** 


10 


4.100 


10 


4.300 


10 

eb w 


4.400 


10 


4.000 


10 


3.800 


10 


3.700* 


\ A 

IV 

10 


3.600* 
3.500** 




3.500** 


10 


4.100 


10 


3.800 


10 


4.200 


10 


4.000 


10 


3.700* 


10 


4.100 


10 

<S> w 


4.000 


10 


3.300** 


10 


4 .000 


10 


3.500** 


10 


3.800 


10 


3.500** 


10 


3.700* 


10 


3 . 600* 


10 


3.900 


10 


3.900 


9 


4.111 


10 


3.400** 


10 


3.500** 


9 


3.778 


10 


3.700* 


AV 


3 • OUU 


8 


3.875 


9 


3.444** 


10 


3.400** 


10 


3.200** 


10 


3.400** 


10 


3.200 


10 


2.700 


10 


2.800 
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Trial ing: Descriptive State/French by Fors 
TOPT Fom-Form C — — — 



N Obs Variable Label 



M 


Mean 


10 


4.000 


10 


4.200 


10 


3.900 


10 


4.600 


10 


3.500. 


10 


10 


3.700* 
3.400 


10 


10 


3.800 


9 


4.556 


10 


4.100 


10 


4.100 


10 


4.200 


10 


3.900 


10 


4.400 


10 


4.000 A 


10 


3.700 


10 


4.300 


10 


4.200 


10 


4.000 


10 


3.600* 


10 


4.000 


10 


4.100 


10 


3.900 


10 


4.000 


10 


4.000 


10 


3.800 


10 


4.400 


10 


4.500 


10 


4.100 


10 


3.900 


10 


4.300 


10 


4.000 


10 


3.700* 


10 


3.600* 


10 


4.100 


10 


3 . 600* 


10 


4.000 


10 


1.900 


10 


3.600 



10 ZTEM1 
XTEM2 
ITEM3 
ITEM 4 
ITEM5 
XTEM6 
XTEM7 
ITEM 8 
ITEM9 
ITEM 10 
ZTEM11 
XTEM12 
XTEM13 
XTEM14 
XTEM15 
XTEM16 
ITEM17 
ITEM 18 
ITEM19 
ITEM20 
XTEM21 
XTEM22 
XTEM23 
XTEM24 
ITEM25 
ITEM26 
ITEM27 
1TEM28 
XTEM29 
XTEM30 
XTEM31 
ITEM32 
XTEM33 
ITEM34 
ITEMS 5 
ITEM36 
ITEM37 
ITEM38 
ITEM39 
ITEM40 



Opening Conversation— General 
Opening Conversation — Ti»e 
Picture #1 — General 
Picture #1— Map 
Picture fl— Tise 
Picture #2— General 
Picture 12 — Picture 
Picture 12 — Tise 
Picture #3— General 
Picture #3— Pictures 
Picture #3— Tise 
Picture #4— General 
Picture 14 — Pictures 
Picturs #4— Tiss 
Picture 15— General 
Picture #5— Pictures 
Picture #5 — Tise 
Topic fl— General 
Topic #1— Tise 
Topic #2— General 
Topic 12 — Ti.Be 
Topic 13— General 
Topic #3 — Tise 
Topic #4 — General 
Topic #4— Tine 
Topic #5— General 
Topic 15— Tise 
Situation II — General 
Situation fl— Tise 
Situation #2 — General 
Situation #2— Tise 
Situation #3— General 
Situation #3— Tise 
Situation 14 — General 
situation 14 — Tise 
Situation #5 — General 
Situation #5— Tine 
Unduly Nervous? 

Replace Target Language Prompt? 
An Accurate Picture? 



ERIC 
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Trial ing: Descriptive Stats/French by For* 
— — — — — TOFT Fore»Fora D — — — — 



N Obs Variable Labal * Maan 



11 ITEN1 


Opening Conversation— General 


11 


4.000 


ITEM2 


Opening Conversation — Tiae 


11 


3.909 


ITEK3 


Picture 41— General 


11 


4.182 


ZTEM4 


Picture #1— Map 


11 


4.455 


ITEMS 


Picture #1— Tiae 


11 


4.091 


ZTEN6 


Picture #2— General 


11 


3.818 


ITEM7 


Picture #2— Picture 


11 


3.909 


ITEM8 


Picture 12 — Tiae 


11 


3.818 


ITEM9 


Picture #3— General 


11 


3.818 


ITEM10 


Picture #3 — Pictures 


11 


4.091 


ZTEN11 


Picture #3— Tiae 


11 


4.O0O 


XTEM12 


Picture #4 — Generel 


11 


4.091 


ZTEM13 


Picture #4— Pictures 


11 


4.091 


ZTEM14 


Picture #4— Tiae 


11 


3.909 


ITEM 15 


Picture #5— General 


11 


3.818 


ZTEM16 


Picture #5— Pictures 


10 


3.700 


ITEM 17 


Picture #5— Tiae 


10 


3.700 


ITEMIB 


Topic fl— General 


11 


3.818 


ITEM19 


Topic 11 — Tiae 


11 


3.545* 


ZTEM20 


Topic #2 — General 


11 


3.909 


ITEM21 


Topic |2 — Tiae 


11 


3.727* 


ITEM22 


Topic #3— General 


11 


3.273** 


ITEM23 


Topic #3 — Tiae 


11 


** 

3.273 


ITEM24 


Topic 14 — General 


11 


3.636* 


ITEM25 


Topic #4 — Tiae 


11 


3.364 


ZTEM26 


Topic #5— General 


11 


4.000 


ZTEM27 


Topic |5 — Tiae 


11 


3.364 


XTEM28 


Situation #1 — General 


11 


3.909 


ZTEM29 


Situation #1— Tiae 


11 


3.636* 


ITEM30 


Situation #2-— General 




J * © JO 

3.727* 


ITEM31 


Situation #2— Tiae 


11 


ITEM32 


Situation #3 — General 


11 


3.909 


XTEM33 


Situation f 3— Tiae 


11 


3.636* 


ITEM34 


Situation #4 — General 


11 


3.091** 


ITEM35 


Situation #4— Tiae 


11 


3.091** 


ITEM36 


Situation #5 — General 


11 


2.909** 


ITEM37 


Situation #5— Tiae 


11 


3.364** 


ITEM38 


Unduly Nervous? 


11 


3.545 


ITEM39 


Replace Target Language Prompt? 


11 


2.545 


ITEM40 


An Accurate Picture? 


11 


3.545 
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Trial ing: Descriptive stata/TOPT-Spanieh by Fori 
— . TOPT Fore-Fora A 



Variabla 


Label 




M 


Mean 


IT EMI 


Opening Conversation—General 


26 


3.692* 


ITEM 2 


Opening Conversation— Tiae 


26 


3.615* 


ITEMS 


Picture i 


11 ""General 

J lew Wf f Vfj W 


26 


3.692* 


ITEM4 


Picture 1 


11— Map 


26 


3.923 


ITEM5 


Picture \ 


U— Tiae 


26 


4.000 


ITEM6 


Picture \ 


12— General 


26 


3.808 


ITEM7 


Picture \ 


12 — Picture 


26 


4.115 


ITEMS 


Picture < 


12 — Tiae 


26 


3.846 


ITEM9 


Picture < 


|3— General 


25 


3.680* 


ITEM10 


Picture i 


13 — pictures 


26 


3 . 692* 


ITEM 11 


Picture i 


13— Time 


26 


3.731* 


ITEM 12 


Picture > 


14— General 


25 


3.760 


ITEM13 


Picture ' 


14 — Pictures 


26 


3.846 


ITEM14 


Picture < 


14— Tiae 


26 


3 . 654* 


ITEM15 


Picture 


15— General 


26 


3.769 


ITEM16 


Picture 


|5— Pictures 


26 


4.077 


ITEM17 


Picture 


15— Tiae 


26 


3.692* 


ITEM18 


Topic #1— General 


26 


3.577* 


ITEM19 


Topic #1 


—Tiae 


25 


3.320** 


ITEM20 


Topic 12 


—General 


25 


3.600* 


ITEM21 


Topic 12 


—Tiae 


25 


3.360** 


ITEM22 


Topic #3 


—General 


25 


3.600* 


ITEM23 


Topic #3 


—Tiae 


25 


3.360** 


ITEM24 


Topic 14 


—General 


26 


3 . 654* 


ITEM25 


Topic #4 


—Tiae 


26 


3.462** 


ITEM26 


Topic #5— General 


25 


3.800 


ITEM27 


Topic #5 


—Tiae 


24 


3.583* 


ITEM28 


Situation #1— General 


26 


3.962 


ITEM29 


Situation #1 — Tiwe 


26 


3.615* 


ITEM30 


Situation #2 — General 


25 


3.760 


ITEM3X 


Situation 12 — Tiae 


26 


3.423** 


ITEM 3 2 


Situation 13— General 


26 


3.808 


ITEM33 


Situation #3 — Tine 


26 


3 . 654* 


ITEM34 


Situation #4 — General 


26 


3.731* 


ITEM35 


Situation I 4 — Tine 


26 


3.269** 


ITEM36 


Situation 15 — General 


26 


3.885 


ITEM37 


Situation 15— Tine 


26 


3.538* 


ITEM38 


Unduly Kervous? 


26 


2.846 


ITEM39 


Replace Target Language Prompt? 


26 


2.846 


ITEM40 


An Accurate Picture? 


26 


3.038 
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Trial ing: Descriptive State/TOPT-Spanieh by Von 
— ~— — TOFT Fora-Fora B — 



N Obs Variable Label 



Mean 



28 



XTEN1 

XTEM2 

ZTEM3 

ITEM4 

ITEMS 

ZTEM6 

ITEM7 

ITEM8 

ITEM9 

ITEM10 

ZTEMX1 

ZTEM12 

ITEM13 

ITEM14 

ITEM15 

ZTEM16 

ZTEM17 

ZTEM18 

ITEM19 

ITEM20 

ITEM21 

ZTEM22 

ITEM23 

ZTEM24 

ZTEM25 

ZTEM26 

ZTEM27 

ZTEM28 

ZTEM29 

ZTEM30 

ZTEM31 

ZTEM32 

ZTEM33 

ZTEM34 

ITEM35 

ZTEM36 

ITEM37 

ITEM38 

ITEM39 

ITEM40 



Opening Conversation— General 
Opening Conversation— Tiae 
Picture #1— General 
Picture #1— Map 
Picture #1 — Tiae 
Picture #2— General 
Picture #2— Picture 
Picture #2 — Tiaa 
Picture #3— General 
Picture #3 — Pictures 
Picture f 3— Tiae 
Picture #4 — General 
Picture #4 — Pictures 
Picture f 4— Tiae 
Picture #5— General 
Picture #5 — Pictures 
Picture #5— Tiae 
Topic #1 — General 
Topic #1 — Tiae 
Topic #2 — General 
Topic #2 — Tiae 
Topic #3 — General 
Topic #3 —Tine 
Topic #4 — General 
Topic 14 — Tiae 
Topic #5 — General 
Topic #5 — Tiae 
Situation fl — General 
Situation fl— Tiae 
Situation 12 — General 
Situation 4 2 --Tiae 
Situation 13 — General 
Situation 43— Tiae 
Situation #4 — General 

14— Tiae 

15 — General 



Situation 
Situation 

Situation 15— Tiae 
Unduly Nervous? 

Replace Target Language Pronpt? 
An Accurate Picture? 



28 
28 

28 

28 

28 

28 

27 

28 

28 

28 

28 

28 

28 

28 

28 

27 

28 

28 

28 

28 

28 

28 

28 

28 

28 

28 

27 

28 

28 

28 

28 

27 

28 

28 

28 

28 

28 

27 

26 

27 



3.857 

3.929 

3.821 

3.893 

3.786 

3.607* 

3.963 

3.500** 

3.714* 

3.643* 

3.429** 

3.929 

3.964 

3.786 

3.786 

3.407** 

3.571* 

3.607* 

3.571* 

3.643* 

3.393** 

3.429** 

3.393** 

3.607* 

3.500** 

3.750 

3.593* 

3.500** 

3.500** 

3.321** 

3.393** 

3.333** 

3.250** 

3.643* 

3.429** 

3.536* 

3.250 

3.074 

2.538 

2.704 
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Trialing: Descriptive State/TOPT-Spanish by Pom 
— — — — TOPT Fora-Porn C — 



N Obs Variable Label 



M 


Mean 


37 


3.568* 


37 




37 


3.486 


38 


3.632* 


38 


3.474** 


37 


3.243** 


38 


3.395** 


38 


3.184** 


38 


3.368** 


38 


4.026 


38 


3.447** 


37 


3.811 


38 


4.132 


38 


3.789 


38 


3.816 


38 


3.816 


38 


3 . 658* 


37 


3.270** 


38 


3.526* 


38 


2.974*** 


38 


3.289** 


38 


2.895** 


38 


2.974** 


38 


3.342** 


38 


3.316** 


38 


3.237** 


38 


3.237** 


38 


3.395** 


38 


3 . 553* 


38 


3 • 5Plr* 


37 


3.568* 


38 


3.447** 


38 


3.316** 


38 


3.053** 


36 


3.111** 


37 


3.405** 


38 


3.395** 


37 


3.405 


38 


3.132 


38 


2.895 



38 ITEM 
XTEM2 
ITEM3 
XTEM4 
XTEM5 
ITEM 6 
XTEM7 
ITEMS 
XTEM9 
ITEM10 
ITEM11 
ITEM12 
ITEM 13 
ITEM 14 
ITEM15 
ITEM16 
ITEM17 
ITEM18 
ITEM19 
ITEM20 
ITEM21 
ITEM22 
ITEM23 
ITEM24 
ITEM25 
ITEM26 
ITEM27 
ITEM28 
ITEM29 
ITEM30 
ITEM31 
ITEM32 
ITEM33 
ITEM34 
ITEM35 
ITEM36 
ITEM37 
ITEM38 
ITEM19 
ITEM40 



Opening Conversation — General 
Opening Conversation — Tine 
Picture il — General 
Picture #1— Map 
Picture 11— Tine 
Picture #2— General 
Picture #2— Picture 
Picture 12 — Tiee 
Picture #3 — General 
Picture #3— Pictures 
Picture #3— Tine 
Picture #4— General 
Picture 14— Pictures 
Picture #4— Tine 
Picture 15— General 
Picture #5— Pictures 
Picture #5— Tine 
Topic #1— General 
Topic II— Tine 
Topic #2— General 
Topic #2 — Tine 
Topic # J— General 
Topic 13 — Tine 
Topic #4 — General 
Topic #4— Tine 
Topic #5 — General 
Topic |5— Tine 
Situation fl— General 
Situation 11— Tine 
Situation #2— General 
Situation #2— Tine 
Situation 13 — General 
Situation #3— Tine 
Situation #4— General 
Situation 14— Tine 
Situation #5 — General 
Situation 15— Tine 
Unduly Nervous? 

Replace Target Language Prompt? 
An Accurate Picture? 



IKS 
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Trialing: Descriptive State/TOPT-Spanish by Pon 
— TOpT f 0rmmTQTU D m 



N Obs Variable Labal 



Mean 



27 



ITEM1 

ZTEN2 

ITEM 3 

ITEM4 

ITEM5 

ITEM 6 

ITEM7 

ITEM 8 

ITEM9 

ITEM 10 

ZTEM11 

ITEM12 

ITEM13 

ITEM14 

ITEM15 

ITEM16 

ITEK17 

ITEM18 

ITEM19 

ITEM20 

ITEM21 

ITEM22 

ITEM23 

ITEM24 

ITEM25 

ITEM26 

ITEM27 

ITEM28 

ITEM29 

ITEM30 

ITEM31 

ITEM32 

ITEM33 

ITEM34 

ITEM35 

ITEM36 

ITEM37 

ITEM38 

ITEM39 

ITEM40 



Opening Conversation— Central 

Opening Conversation — Ties 

Picture #1— General 

Picture il — Map 

Picture #1— Ties 

Picture #2— General 

Picture #2— Picture 

Picture #2— Tiee 

Picture #3— Generel 

Picture #3— Pictures 

Picture #3— Tiee 

Picture #4 "Generel 

Picture #4— Pictures 

Picture #4— Tiee 

Picture 15— General 

Picture #5— Pictures 
Picture #5— Tiee 
Topic #1 — General 
Topic #1— Tiee 
Topic #2" General 
Topic #2— Tiee 
Topic 13— General 
Topic #3— Tiee 
Topic f 4 — General 
Topic #4— Tiee 
Topic #5— General 
Topic #5— Tiee 
Situation #1— General 
Situation 11 — Tiee 
Situation #2— General 
Situation #2— Tiee 
Situation #3— General 
Situation 13— Tiee 
Situation f4 — General 
Situation #4— Tiee 
Situation 15 — General 
Situation #5— Tiee 
Unduly Nervous? 

Replace Target Language Prompt? 
An Accurate Picture? 



27 

26 

27 

27 

26 

26 

27 

27 

26 

27 

27 

26 

27 

27 

26 

27 

27 

25 

27 

27 

27 

26 

27 

26 

26 

27 

27 

26 

26 

27 

27 

26 
26 
27 
27 
26 
26 
24 
27 
26 



** 



3.963 
4.077 
4.000 
4.037 
4.038 
3.423** 
3.815 
3.185 
3.885 
4.185 
3.852 
4.115 
4.222 
3.778 
3.692* 
3.593* 
3.519* 
3.720* 
3.815 

3.889 

3.778 

3.577* 

3.630* 

3.577* 

3.577* 

3.926 

3.704 

4.038 

3.846 

3.778 

3.704* 

3.654* 

3.577* 

3.815 

3.556* 

3.923 

3.769 

3.292** 

2.704 

3.154 
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APPENDIX H 



EXAMPLE OF PRINTOUT OF TOTALING EXAMINEES' WRITTEN COMMENTS 

ON INDIVIDUAL ITEMS 
(Example from TOPT-Spanish, Form B, Picture 4) 
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4520843 Spanish 21 Paso 



SPANISH B p4 



Humorous. Unclear at first but clsars up as you 
go on to ths rast of tha pieturas 



4523716 bilingual education Edinburg SPANISH B p4 

accurate picture; response a bit slow 



4553315 bilingual education Edinburg SPANISH B p4 

This example vas quite clear but I seemed to get 
tongue tied with my vocabulary. I also felt there 
should be a little sore time allowed. 



4554973. Spanish Edinburg SPANISH B p4 

Just a little bit confusing because if you see 
pictures 3 and 4 on that page, the owner and the 
second guy get mixed up because they have the same 
color shirt. 
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APPENDIX I 



MEAN ITEM QUALITY RATINGS 
FROM THE TRIALING JUDGES' RESPONSE SHEET 

(By Language and Form) 
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Trial ing: French Quantity Ratings by For* 
— — TOPT Pora-form A — — — — — « 



Oba 


Variable 


11 


Haan 


10 


CQUAL 


6 


2.66667 




P1QUAL 


7 


2.57X43 




P2QUAL 


7 


2.71428 




P3QUAL 


8 


2.75000 




P4QUAL 


8 


2.50000 




P5QUAL 


8 


2.37500 




T1QUAL 


8 


2.75000 




T2QUAL 


8 


3.00000 




T3QUAL 


8 


2.75000 




T4QUAL 


7 


2.85714 




T5QUAL 


7 


2.85714 




S1QUAL 


8 


2.37500 




S2QUAL 


8 


2.62500 




S3QUAL 


8 


2.75000 




S4QUAL 


7 


2.71429 




S5QUAL 


8 


2.75000 



TOPT Form-Form B 



Variable 


N 


Mean 


CQUAL 


8 


1.87500 


P1QUAL 


9 


2.22222 


P2QUAL 


9 


2.22222 


P3QUAL 


9 


2.00000 


P4QUAL 


8 


2.50000 


P5QUAL 


9 


2.22222 


T1QUAL 


8 


2.50000 


T2QUAL 


9 


1.88889 


T3QUAL 


8 


2.37500 


T4QUAL 


9 


2.33333 


T5QUAL 


9 


2.33333 


S1QUAL 


9 


2.22222 


S2QUAL 


8 


2.00000 


S3QUAL 


8 


2.50000 


S4QUAL 


8 


2.25000 


S5QUAL 


8 


2.37500 
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Trial ings Franch Quantity Ratings by Fora 
TOPT Fora-Fora c ——->——. 



N Obs Variable N Maan 



10 CQuAL 


9 


2.77778 


P1QUAL 


8 


•a naaaa 

2 a 75000 


P2QUAL 


8 


2 # 75000 


F3QUAI* 


8 


3*00000 


P4QUAL 


8 


3,00000 


P5QUAX 


8 


3.00000 


T1QUAL 


8 


3.00000 


T2QUAL 


7 


3.00000 


T3QUAL 


7 


3.00000 


T4QUAL 


8 


3.00000 


T5QUAL 


8 


2.87500 


S1QUAL 


8 


2.75000 


S2QUAL 


7 


3.00000 


S3QUAL 


8 


3.OC0O0 


S4QUAL 


7 


3.00000 


S5QUAL 


7 


3.00000 



TOPT Forn-Fora D 



Variable 


N 


Mean 


CQUAL 


9 


2.77778 


P1QUAL 


9 


2.88889 


P2QUAL 


9 


3.00000 


P3QUAL 


7 


2.85714 


P4QUAL 


9 


2.88889 


P5QUAL 


9 


2.77778 


T1QUAL 


8 


3.00000 


T2QUAL 


8 


3.00000 


T3QUAL 


8 


3.00000 


T4QUAL 


9 


2.77778 


T5QUAL 


8 


2.87500 


S1QUAL 


6 


2.66667 


S2QUAL 


6 


2.83333 


S3QUAL 


6 


2.83333 


S4QUAL 


6 


2.66667 


S5QUAL 


6 


3.00000 
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Trialing: Spanish Quantity Ratings by Fora 
... TOPT Fom-Fora A — — — — -™. 



N Obs Variabla 



Mean 



26 CQUAL 


21 


2*57143 


P1QUAL 


20 


2.65000 


P2QUAL 


22 


2.68182 


P3QUAX* 


20 


2.70000 


DlfMtlT 


19 


2.7B947 


P5QUAL 


22 


2.77273 


T1QUAL 


20 


2.65000 


T2QUAL 


22 


2.77273 


T3QUAL 


21 


2.76190 


T4QUAL 


22 


2.77273 


T5QUAL 


21 


2.76190 


S1QUAL 


22 


2.72727 


S2QUAL 


20 


2.75000 


S3QUAL 


19 


2.73684 


S4QUAL 


21 


2.76190 


S5QUAL 


20 


2.80000 



TO°T Fora-Fona B 



N Obs Variabla 



N 



He an 



28 CQUAL 


18 


2. 


61111 


P1QUAL 


18 


2. 


50000 


P2QUAL 


18 


2. 


61111 


P3QUAL 


17 


2. 


58824 


P4QUAL 


18 


2. 


77778 


P5QUAL 


19 


2. 


63158 


T1QUAL 


19 


2. 


52632 


T2QUAL 


20 


2. 


70000 


T3QUAL 


20 


2. 


35000 


T4QUAL 


20 


2. 


65000 


T5QUAL 


20 


2. 


55000 


S1QUAL 


20 


2. 


65000 


S2QUAL 


20 


2. 


65000 


S3QUAL 


19 


2. 


36842 


S4QUAL 


19 


2. 


52632 


S5QUAL 


19 


2. 


68421 
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a ling: Spanish Quantity lutings by 
™— — TOPT Por*-Por» C 



N 0b» 


Variable 


N 


Mean 


38 


CQUAL 


19 


2.57895 




PXQUAL 


21 


2.38095 




P2QUAL 


20 


2.55000 




P3QUAL 


21 


2.66667 




P4QUAL 


21 


2.66667 




P5QUAL 


20 


2.60000 




T1QUAL 


21 


2.57143 




T2QUAL 


20 


2.60000 

ww w Www w 




T3QUAL 


21 


2.38095 




T4QUAL 


21 


2 . 66667 




T5QUAL 


21 


2.47619 




S1QUXL 


21 


2 .66667 

ww w ww w9 %p wr w 




S2QUAL 


21 


2.57143 




S3QUAL 


21 


2.61905 




S4QUAL 


20 


2.50000 




S5QUAL 


21 


2.66667 














N Obs 


VariabXa 


N 


Mean 


0% mm 

27 


CQUAL 


19 


2 • 68421 




P1QUAL 


20 


2 #70000 




P2QUAL 


19 


0% mt n m *% w. 

2 . 68421 




P3QUAL 


19 


0% AMA JM 

2 .78947 




P4QUAL 


20 


*■> AC AAA 

2 .85000 




P5QUAL 


20 


2.70000 




T1QUAL 


21 


2.71429 




T2QUAX 


20 


2.70000 




T3QUAL 


21 


2.61905 




T4QUAL 


20 


2.65000 




T5QUXL 


21 


2.61905 




S1QUAL 


21 


2.76190 




S2QUXL 


21 


2.71429 




S3QUAL 


20 


2.55000 




S4QUAL 


21 


2.76190 




S5QUAL 


19 


2.78947 
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APPENDIX J 



EXAMPLE OF TRIALING JUDGES* QUANTITATIVE TIME DATA BY 

FORM AND ITEM 
(Example from TOPT-Spanish, Form A, 
Opening Conversation Items 9 and 10, 
and Picture Item 1) 
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Trial ing: Spanish Tiate by Fora 
........ topt Fora-Fora A — — — 



_ Cuaulativa Cuaulativa 

2122 Pwrc#nt frequency Percent 

-2*5 i » 22.7 

? 14 63.6 21 95.5 

2 ' 5 1 4.5 22 100.0 

Frequency Kissing - 4 



C10TIME 


Frequency 


Percent 


Cumulative 
Frequency 


Cumulative 
Percent 


-15 




4.5 


1 


4.5 


-7.5 




4.5 


2 


9.1 


-5 




27.3 


8 


36.4 


-2.5 




13.6 


11 


50.0 


0 


11 


50.0 


22 


100.0 



Frequency Kissing - 4 



Cuaulativa Cumulative 
P1TXKE Frequency Percent Frequency Percent 



-7.5 


1 


4.5 


1 


4.5 


0 


11 


50.0 


12 


54.5 


5 


2 


9.1 


14 


63.6 


7.5 


3 


13.6 


17 


77.3 


10 


1 


4.5 


18 


81.8 


12.5 


4 


18.2 


22 


100.0 



Frequency Missing - 4 
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APPENDIX K 



EXAMPLE OF TRIALING JUDGES' COMMENTS ON INDIVIDUAL 

EXAMINEE RESPONSES 
(TOPT-Spanish, Form B) 
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2824699 



SPANISH B 



Stansfield 
2.0 

pi +10 3 



4507701 

Tisnado 
2.0 

pi +7.5 2 



SPANISH B 



3716290 



SPANISH B 



Downey 
1.90 

pi 0 1 



Gave directions as if the nan who asked the 
question were looking at the nap also. Used 
words like "right here" as if he were 
pointing to something on the aap. 



4553315 

Tisnado 
1.90 

pi- •'•7. 5 2 



SPANISH B 



4576537 



SPANISH B 



Marc Ferrara 
1.80 

pi 0 



Addresses executives as "tu". Distinction 
between "derecha" and "derecho* unclear. 
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4661702 



SPANISH B 



Downey 
2.0 

p2 +15 1 



Didn't give any details and bad lota of extra 
tine. Maybe should include "Include es many 
details as possible 1 * in the instructions. 



2824699 

Stansfield 
2.0 

p2 



SPANISH B 



3716290 

Downey 
1.90 

p2 0 



SPANISH B 



455331598 



SPANISH B 



Tisnado 
1.90 

P2 



She would have needed auch more tine since 
she did not follow instructions properly. 
She started describing "houses" more than 
activities at American hones. 



4576537 



SPANISH B 



Marc Ferrara 
1.80 

p2 0 



la "estova" for stove, "laapae" for lamp 



o 
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4503904 
Tisnado 



SPANISH 8 



■3 



4649411 

Downey 
(missing) 

■3 -5 



SPANISH B 



4634975 

Stansf ield 
4 



S3 



4-10 3 



SPANISH A 



Student did not nans a place. Instead, she 
cited advantages of school trips. 



4576868 

Stansf ield 
3.0 

S3 0 



4635590 

Bass 
3 



63 



-10 



SPANISH A 



Proposes to take group to Danals Supermarket 
in Dallas, where they will learn about 
Mexican food * customs. 

SPANISH A 



4590691 

Stansfield 
3.0 

S3 0 



SPANISH A 



E invited Isabel to go with him to San 
Antonio 
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APPENDIX L 



INSTRUCTIONS AND RESPONSE SHEETS FOR THE 
CONTENT VALIDATION COMMITTEES 



183 



(for Content Validation Study) 

TOPT ITEM JUDGEMENT SHEET INSTRUCTIONS 
You have been given a booklet containing the items from the 
three forms of the Texas Oral Proficiency Test (TOPT) for Spanish 
or French, three copies of the Judge's Response Sheet, and a 
yellow form with an example of how to use the Judge's Response 
Sheet. On each form of the test are 15 items classed into three 
groups: picture items, topic items and situation items. Each 
group contains five items. In your item booklet are the English 
item instructions (English prompts) and the native language 
prompt, to which the examinees respond. This is followed by two 
sets of times? the first indicates the amount of time the 
examinees have to prepare a response and the second indicates the 
amount of time the examinees have to give the response. The 
picture items are followed in your item booklet by the actual 
pictures the examinees see in their test booklets. 

The TOPT is administered in a language lab. In the actual 
testing situation, examinees hear all instructions, including 
both prompts, through their headphones. They also have a test 
booklet, which contains the general instructions to the test, the 
English prompts, the time limits, and the pictures. When they 
take the test, they read the English prompt in their test 
booklets while listening to it being read from a master tape, 
prepare their response during the time indicated, hear the native 
language prompt (which is NOT printed in the test booklet) , and 
then respond during the time allotted. The master tape paces the 
test. Each individual's responses are recorded on a separate 
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tape, which is later scored by trained raters. 

Each of the TOPT items has an associated speaking task, such 
as giving directions or narrating in past time. These tasks are 
indicated in both the item booklet you have received and on the 
Judge's Response Sheet. Your charge, as a member of the content 
validation study committee, is to decide whether, in your 
professional judgement, each item matches the speaking task 
associated with it. If you decide yes, then you circle -Yes* in 
the corresponding place on the Judge's Response Sheet and move on 
to the next item. If you decide no, you circle -Ho" in the 
corresponding place on tht Judge's Response Sheet AND write in 
the space tor that item the reasons you believe that the item 
does NOT match the speaking task associated with it. It is 
extremely important that you document each negative response with 
the reason or reasons you believe there is an inappropriate fit. 

You have received three copies of the Judge's Response 
Sheet, one for each of the three TOPT forms. On each one write 
your name, the date, and the TOPT form you are evaluating with 
that Judge's Response Sheet. 

We will now do an example together, taken from one of the 
TOPT test forms, which is printed on the first page of the 
Judge's Response Sheet. Please read the example now. 
(Judges read example.) 

(Generate group discussion and consensus.) 
If you have no further questions, let us begin. Please 
write your name and the date or. the first Judge's Response Sheet. 
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Then write number "241" for the form number. The question you 
ere to consider is printed et the top of your Judge* s Response 
Sheet: "Does the item match the speaking task associated with 
it?" Now please open the item booklet and begin reading the 
items . 
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EXAMPLE OP HOW TO USE THE JUDGE'S RESPONSE SHEET 
TOPT Content validation Study — FRENCH 



The following example will show you how to use the Judge's Response 
Sheet. 

A. Below is a sample item from the teat as presented in the item 
booklet. Now read the item and progress to part B below. 



Situation 5: GIVE A BRIEF FACTUAL SUMMARY 

A group of high school student* has arrived from France to spend 
a summer session at your local community college. You have been 
asked to give then a brief talk on sose important recent local or 
national eventa. You say talk about events in politics, economics, 
education, sports, etc. After your talk is introduced, brief the 
group en sose major local or national eventa that have occurred 
recently. 

** Maintenant, votre guide va vous dormer des informations sur 
quelques evenements r Scents. [Male voice) 

(30 sec / l ain 35 eecji 

B. On the Judge 'a Response Sheet, circle YES if you feel that 
the item matches the speaking task, or circle NO if you feel 
that the item does not match the speaking task AND WRITE YOUR 
REASONS FOR FEELING THIS WAY. Now indicate below whether or 
no you think Topic 3 matches the speaking task associated with 
it. 



TOPT Content Validation Study 
» TOPT-French 
Judge's Response Sheet 



Item/Judgment Speaking Task/Comments 



SITUATION 5 GIVE A BRIEF FACTUAL SUMMARY 
Yes No 
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EXAMPLE OF HOW TO USE THE JUDGE'S RESPONSE SHEET 
TOFT Content Validation Study — SPANISH 



The following exaaple will ahov you hoy to uaa the Judge's Response 
Sheet. 

A. Below is a sample itea froa the test as presented in the itea 
booklet. Now read the itea and progress to part B below. 



Topic 3: GIVE A BRIEF FACTUAL SUMMARY 



A group of high school students has arrived froa Colombia to spend 
a summer session at a community college in Texas. You have been 
asked to give a brief talk as part of their orientation on two or 
three recent events in Texaa that you feel are important. After 
your talk ia introduced, brief the group on theae recent events . 

** Y ahora, como parte de nuestra orientaci6n, vaaos a escuchar 
una charla sobre algunos acontecimientos recientes an el 
estado de Tejaa. [Female voice) 

(30 sec / 1 min 30 sec) 



B. On the Judge's Response Sheet, circle YES if you feel that 
the item matches the speaking task, or circle NO if you feel 
that the item does not match the speaking task AND WRITE YOUR 
REASONS FOR FEELING THIS WAY. Now indicate below whether or 
no you think Topic 3 matches the speaking task associated with 
it. 



TOPT Content Validation Study 
TOPT-Spanish 
Judge's Response Sheet 



It em/ Judgment Speaking Task/Comments 



TOPIC 3 



GIVE A BRIEF FACTUAL SUMMARY 
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TOPT Content Validation Study 
TOPT-Freneh 
Judge's Response Sheet 

Name: 

Dates — ■ — 

TCPT Form: 

Does the item match the speaking task associated with it? 

Item/Judgment Speaking Task/Comments 

Picture 1: GIVE DIRECTIONS 
Yes No 



Picture 2: DESCRIBE A PLACE/ ACTIVITIES 

Yes No 



Picture 3: NARRATE IN PRESENT TIME 
Yes No 



Picture 4: NARRATE IN PAST TIME 

Yes No 



Picture 5: NARRATE IN FUTURE TIME 

Yes No 



lbn 



Topic 1: 
Yes No 



DESCRIBE PERSONAL ACTIVITIES 



9 

ERIC 



Topic 2! STATE k VANTAGES/ DISADVANTAGES 

Yes No 



Topic 3: GIVE INSTRUCTIONS 

Yes No 



Topic 4: SUPPORT AN OPINION 

Yes No 



Topic 5: HYPOTHESIZE ON A PERSONAL TOPIC 

Yes No 
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Situation l: MAKE SIMPLE REQUESTS 
Yes No 



Situation 2 : GIVE ADVICE 
Yes No 



Situation":}! SPEAK WITH TACT (e.g. Apologize, Lodge a Complaint) 
Yes No 



Situation 4; STATE PERSONAL POINT OF VIEW 
Yes No 



Situation 5: GIVE A BRIEF FACTUAL SUMMARY 
Yes No 
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TOFT Content Validation Study 
TOPT-Spaniah 
Judge* a Response Sheet 

Name: 

Date: __ZIZ_Z^ZZ_ZZ___^ZZ__Z_ZIZZ_^ 

TOPT Form: 

Does the item Batch the speaking task associated with it? 
Item/Judgment Speaking Task/Comments 
Picture 1: GIVE DIRECTIONS 
Yes . No 



Picture 2: DESCRIBE A PLACE/ACTIVITIES 

Yes No 



Picture 3: NARRATE IN PRESENT TIME 

Yes No 



Picture 4: NARRATE IN PAST TIME 

Yes No 



Picture 5: NARRATE IN FUTURE TIME 

Yes No 
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Topic 1: GIVE INSTRUCTIONS 

Yes No 



Topic 2: STATE ADVANTAGES/DISADVANTAGES 

Yes No 



Topic 3: GIVE A BRIEF FACTUAL SUMMARY 

Yss No 



Topic 4: SUPPORT AN OPINION 

Yes No 



Topic 5: HYPOTHESIZE ON AN IMPERSONAL TOPIC 

Yes No 



0 

ERIC 



193 



Situation 1: 
Yes No 



SPEAK WITH TACT (e.g. Apologize, Lodge a Conplaint) 



Situation 2: SPEAK TO PERSUADE SOMEONE 
Yea No 



Situation 3: PROPOSE AND DEFEND A COURSE OF ACTION 
Yes No 



Situation 4: GIVE A PROFESSIONAL TALK 
Yes No 



Situation 5: GIVE ADVICE 
Yes No 
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(for Content Validation Study —Bilingual Education) 

(NOTE: These vere sllgh-ly revised for the bilingual education 

study following experience with the French and Spanish Study. ) 

TOPT ITEM JUDGEMENT SHEET INSTRUCTIONS 

You have been given a booklet containing the items froa the 

three forms of the Texas Oral Proficiency Test (TOPT) for 

Spanish, three copies of the Judge's Response Sheet, and a 

colored form with an example of how to use the Judge's Response 

Sheet. On each form of the test are 15 items classed into three 

groups: picture items, topic items and situation items. Each 

group contains five items. In your item booklet are the English 

item instructions (English prompts) and the native language 

prompt, to which the examinees respor.d. This is followed by two 

sets of times; the first indicates the amount of time the 

examinees have to prepare a response and the second indicates the 

amount of time the examinees have to give the response. The 

picture items ere followed in your item booklet by the actual 

pictures the examinees see in their test booklets. 

The TOPT is administered in a language lab. In the actual 

testing situation, examinees hear all instructions, including 

both prompts, through their headphones. They also have a test 

booklet, which contains the general instructions to the test, the 

English prompts, the time limits, and the pictures. When they 

take the test, they read the English prompt in their test 

booklets while listening to it being read from a master tape, 

prepare their response during the time indicated, hear the native 

language prompt (which is NOT printed in the test booklet) , and 

then respond during the time allotted. The master tape paces the 
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test. Each individual'* response* are recorded on a separate 
tape, which is later scored by trained raters. 

L*ch of the TOPT items has an associated speaking task, such 
as giving t ; - Tactions or narrating in past ties. These tasks are 
indicated in both the item booklet you have received and on the 
Judge's Response Sheet. Your charge, as a member of the content 
validation study committee, is to decide whether, in your 
professional judgement, each item matches the speaking task 
associated with it. If you decide yes, then you circle "Yes* in 
the corresponding place on the Judge's Response Sheet and move on 
to the next item. If you decide no, you circle "No" in the 
corresponding place on the Judge's Response Sheet AND write in 
the space for that item the reasons you believe that the item 
does NOT match the speaking task associated with it. It is 
extremely important that you document each negative response with 
the reason or reasons you believe there is an inappropriate fit. 

You have received three copies o'£ the Judge's Response 
Sheet, one for each of the three TOFT forms. On each one please 
write your name and the date now. Now on one of the sheets 
please write 182 for the TOPT form number. On the second sheet 
write 241 for the form number and on the last write 663. 

(Wait for judge's to fill in information) 

He will now do three examines together with items similar to 
those on one of the TOPT test forms, which are printed on the 
colored paper. Please read and do these examples now. When you 
have completed them, we will discuss these examples. 

(Judges read example and complete response form.) 



e 
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(Generate group discussion and consensus.) 
******* Notes for the Proctor on the example items. *********** 

Example 1. There ehould be consensus that the item fits the 
task. Thie one was the only sample item for the French and 
Spanish committee and there was no disagreement and no 
discussion. 



Example 2. There should be consensus that the item DOES NOT fit 
the task. This item is actually for •Give Instructions." 

Example 3. This is an example of an item where the item Batches 
the task, but some aspects of the item may cause some to say the 
item does not match the task. It is illogical that the examinee 
speaks to a hotel manager in Spanish in Dallas. Also, 35 seconds 
is much too short to carry out the task. However, point out that 
since the item does match the speaking task (concentrate on the 
bold-faced part of the prompt), the answer should be YES. 
Mention that they may find aspects of an item that they may feel 
could have been done in a different way, but that does not 
necessarily mean that the item does not match the task associated 
with it. They need to continually focus on the question at hand: 
Does the item match the speaking task associated with it? Remind 
them that all the items they read have been through various 
committees and have been "field tested. 

If you have no further questions, let us begin. Please make 
sure your name and the date are written on the first Judge's 
Response Sheet, and that the form number is "182." The question 
you are to consider is printed at the top of your Judge's 
Response Sheet: "Does the item match the speaking task 
associated with it?" Now please open the item booklet and begin 
reading the items. 
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TOPT content Validation Study 
TOPT-Bi lingual Education 
Judge * s Response Sheet (Example) 



Name: 
Date: 



TOPT Form: Example Items 

Does the item match the speaking task associated with it? 

Item/Judgment Speaking Task/Comments 
EXAMPLE 1 GIVE A BRIEF FACTUAL SUMMARY 

Yes NO 



EXAMPLE 2 
Yes No 



PROPOSE AND DEFEND A COURSE OF ACTION 



EXAMPLE 3 
Yes No 



SPEAK WITH TACT (e.g. Apologize, Lodge a Complaint) 
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EXAMPLE OF HOW TO USE THE JUDGE'S RESPONSE SHEET 
TOPT Content Validation Study — BILINGUAL EDUCATION 



Below are three saaple items froa the TOFT as presented in the item 
booklet. Mark your responses to these examples on the example 
Judge's Response sheet attached. Remember these instructions: 
Circle YES if you feel that the item matches the speaking task. 
Circle NO if you feel that the itea does not match the speaking 
task AND WRITE YOUR REASONS FOR FEELING THIS WAY. 



w.n m<nn'iu m nw yi » MUI^ — . pi ■ m m t m i .mu mi m_ l _l_M. ii— h i ■ in .i i . ,ji .. 



Example It GIVE A BRIEF FACTUAL SUMMARY 

A group of high school students has arrived froa Colombia to spend 
a summer session at a community college in Texas. You have been 
asked to give a brief talk as part of their orientation on two or 
three recent events in Texas that you feel are important. After 
your talk is introduced, brief the group oa these recent events. 



** 



Y ahora, como parte de nuestra orientacidn, vanos a escuchar 
una charla sobre algunos acontecinientos recientes en el 
estado de Tejas. [Female voice] 

(30 sec / l ain 30 sec) 

Example 2: PROPOSE AND DEFEND A COURSE OF ACTION 

An exchange teacher froa Venezuela, Mr. Medina, has come to you 
for information on the first day of school. Re will be teaching 
American students for «Ae first tine, and he would like to know 
how roll cell is conducted in American classrooms. After Mr. 
Medina asks his question, briefly explain to hia the procedure for 
taking attendance in a typical Aaerican classroom. 

** £C6mo se pasa lists en las escuelas norteamericanas? 
(15 sec / l min) 

Example 3: SPEAK WITH TACT (e.g. ApologiEerL^grrco_plain_r" 

Yo. are leading a group of Spanish-speaking students on a tour of 
Texas. When you arrive et e hotel in Dallas, where you had already 
paid a deposit, the clerk tells you there ere no rooms available. 
Y ? u to speak with the manager, Mr. Navarro. After he asks you 
what the problem is, sxplain the situation to him. ask him to 
remedy it, conveying both your feelings about what has happened and 
your urgent need to find accommodations for the group. 

** Buenos dias. Me dijo el recepcionista que queria hablar 
conmigo. iZn que puedo servirle? 

(15 sec / 35 sec) 
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APPENDIX M 



INSTRUCTIONS AND RATING SHEETS 
FOR TEXAS ACTFL RATERS 
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Instructions to Raters on Completing the Rater's Evaluation Sheet 



Thank you for accepting the important task of rating the 
tapes that will be used to select segments for the standard 
setting study. 

Enclosed are the tapes you are to rate, an oath of 
confidentiality, and a number of other materials that are 
mentioned on the checklist. After you have gone through all of 
this, you should begin rating tapes based on Form A. You will 
note that we have included copies of a Rater's Evaluation Sheet 
that is designed to assist you in the process of making a rating. 
Please complete all the information at the top of this sheet as 
you rate each examinee. 

In general, we want you to perform two main tasks: assign a 
rating to each examinee on the 15 prompts that follow the warm up 
and assign an overall rating to each examinee. In an OPI, the 
warm-up and the wind-down do not play a significant role in 
determining the examinee's rating. The ra* 4 ng is based primarily 
on the level check and the probe phases. \ ms, you do not have 
to listen to the warm-up (opening conversation) here. However, 
in the case of an examinee who scores at the intermediate level, 
it may be useful to go back and listen to the warm-up before 
making an overall rating. This is because the warm-up includes 
questions at the intermediate level, and the responses to these 
questions may help you decide between an Intermediate Low and an 
Intermediate Mid, for example. If you do listen to the warm-up 
for an examinee, please assign a rating for the warm-up at the 
bottom of page one of the Rater's Evaluation Sheet. 

The TOPT Rater's Evaluation Sheet is one that we have 
drafted for use in the rating of TOPT operational test tapes. We 
believe it will be helpful to you also. While you may wish to 
complete all the information requested for each prompt, we will 
be using your "Rating for this item" only, which is located at 
the bottom of the space devoted to each item on the Rater's 
Evaluation Sheet. Please circle the appropriate abbreviation of 
the ACTFL levels. 

You will notice that the highest level is S+ (High 
Superior) . This level is equivalent to levels 3+ to 5 on the ILR 
scale. It would be assigned to an examinee whose performance 
suggests that he or she is at this level (i.e., clearly above a 
Superior) . 

Please be aware that the Rater's Evaluation Sheet forms on 
which you are to record your ratings are for the final version of 
the TOPT. Thus, they reflect minor changes in the tests made 
after trial ing (with the exception of the deletion of the social 
security number from the opening conversation). Accordingly, you 
may notice very minor differences in the topic description for 
some items. Functions, however, will always be the same. You do 
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not need to bring these differences to our attention. If you 
have any doubts about what to do, or about how to rate a specific 
examinee, please give me a call. 

Before you begin rating, you may wish to "recalibrate" 
yourself to an external criterion. As an anchor point for these 
ratings, we will use the examples of four proficiency levels 
included in the Lisken-Gasparro familiarization Kit. These 
ratings were widely verified by other raters and tester trainers 
before being included in the kit. Thus, they serve as a useful 
anchor for several points on the scale. 

You will note that we have asked you to rate the tapes in a 
specific order, and to forward us the tapes and ratings that you 
have completed as of September 24th. This will allow us to begin 
inputting the data, while you continue to score tapes. He are 
under a very tight schedule for this project, so please stay on 
schedule. It will undoubtedly mean that you will have to devote 
a large part of your weekend to this task. 

All of us associated with this project are thankful to have 
the cooperation of dedicated professionals like yourself. 
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TOFT Rater's Evaluation Sheet 
FRENCH TRIAL FORM B 



Name of Examinee 

Social Security Number 



Rater's Name 
Date of Rating 



Examinee Level Rating 
Rating Verified By 



Opening Conversation: 
1. Name 



2. Social Security Number 



3. Place where living 



4. Describe place where living 



5. Preferred activities in primary scliool 



6. Influential teacher in secondary school 



1. Decision to teach French 



8. Preferences in teaching French culture 



9. Use of French outside of school 



10. Wliy French is liked 



OVERALL RATING THUS FAR: N 1L IM IH A A+ S S+ 
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Picture #1 Function: Give directions Topic: From bookstore to restaurant 

Successfully Handled: Yes No 

CftPteptflg'tfig Accuracy 

Strengths: 
Problems: 

Rating for this item: N IL IM IH A A+ S S+ 



Picture #2 Function: Describe a place/activities Topic: Typical American home 
Successfully Handled: Yes No 

CBPlatifltoj, Accuracy 

Strengths: 
Problems: 

Rating for this item: N IL IM IH A A+ S S+ 



Picture #3 Function: Narrate in present tense Topic: School librarian 

Successfully Handled: Yes No 

Content/Ideas Accuracy 

Strengths: 
Problems: 

Rating for this item: N IL IM IH A A+ S S+ 



Picture #4 Function: Narrate in past tense Topic: Dry cleaners 

Successfully Handled: Yes No 

Content/Ideas Accuracy 

- Strengths: 
Problems: 

Rating for this item: N IL IM IH A A+ S S+ 



Picture #5 Function: Narrate in future tense Topic: Surprise kinfulay party 

Successfully Handled: Yes No 

Cpptent/Ifleas Accuracy 

Strengths: 
Problems: 

Rating for this item: N IL IM IH A A+ S S+ 
OVERALL RATING THUS FAR: N IL IM IH A A+ S S+ 
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Tod'c #1 Function: Describe personal activities Topic: Places to see in the Southwest 
Successfully Handled: Yes No 

Content/Ideas Accuracy 

Strengths: 



Problems: 

Rating for f Ms item: 



N IL IM IH A A+ S S+ 



Topic #2 Function: State advantages/disadvantages Topic: Puviic transportation 
Successfully Handled: Yes No 

Content/Ideas Accuracy 

Strengths: 



Problems: 

Rating for this item: 



N IL IM IH A A+ S S+ 



Topic #3 Function: Give instructions 
Successfully Handled: Yes No 
Content/Ideas 

Strengths: 



Topic: Taking attendance 



Accuracy 



Problems: 

Rating for this item: 



N IL IM IH A A+ S S+ 



Topic #4 Function: Support an opinion 
Successfully Handled: Yes No 
Content/Ideas 

Strengths: 



Topic: Language error correction 



Accuracy 



Problems: 

Rating for this item: 



N IL IM IH A A+ S S+ 



Topic #5 Function: Hypotliesize on a personal topic Topic: Retirement 
Successfully Handled: Yes No 

COntcnt/Ideas Accuracy 

Strengths: 



Problems: 

Rating for this item: 



N IL IM IH A A+ S S+ 



OVERALL RATING THUS FAR: N IL IM IH A A+ S S+ 
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Situation #1 Function: Make simple requests Topic: Car rental 

Successfully Handled: Yes No 

ggltttfllflteg Accuracy 

Strengths: 



Problems: 

bating for this item: N IL IM IH A A+ S S+ 



Situation #2 Function: Give advice Topic: Participating in exchange program 

Successfully Handled: Yes No 

Content/Ideas Accuracy 

Strengths: 



Problems: 

Rating for this item: N IL IM IH A A+ S S+ 



Situation #3 Function: Speak with tact Topic: Complain to hotel manager 

Successfully Handled: Yes No 

Content/Ideas Accuracy 

Strengths: 
Problems: 

Rating for this item: N IL IM IH A A+ S S+ 



Situation #4 Function: State personal point of view Topic: U.S. government spending 
Successfully Handled: Yes No 

Content/Ideas Accuracy 

•Strengths: 



Problems: 

Rating for this item: 



N IL IM IH A A+ S S+ 



Situation #5 Function: Give a brief summary 
Successfully Handled: Yes No 
Content/Ideas 

Strengths: 



Topic: Recent local or national events 
Accuracy 



Problems: 

Rating for this item: 



N IL IM IH A A+ 



S+ 



OVERALL RATING ON TEST: N IL IM IH A A+ S 

-Please make any additional comments here— 



S+ 
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TOFT Hater's Evaloatfcn Sheet 
SPANISH TRIAL FORM B 



Name of Examinee 

Soda] Security Number 

Examinee 1 ! Teaching Area: Spanish Bilingual Education 

Rater*t Name 

Date of Rating 

Examinee Level Rating 

Rating Verified By 

Opening Conversation: 

1. Name 

2. Social Security Number 

3. PLtce when living 



- 4. Describe place where living 



5. Preferred activities in primary school 



6. Influential teacher in secondary school 



7. Interest in teaching 



8. Experience in teaching 



9. Enjoyment in speaking Spanish 



10. Opportunities to use Spanish outside of school 

OVERALL RATING THUS FAR: N EL IM IH A A+ S S+ 
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Picture #1 Function: Give directions Topic: From bookstore to restaurant 

Successfully Handled: Yes No 

CcnkntfldcM issma 

Strengths: 
Problems: 

Rating for this item: N IL IM IH A A+ S S+ 



Picture #2 Function: Deserve a place/activities Topic: typical American home 
Successfully Handled: Yes No 

CmtenlflMfla Asamsx 

Strengths: 
Problems: 

Ratios for this item: N IL IM IH A A+ S S+ 



Picture #3 Function: Narrate in present tense Topic: School librarian 
Successfully Handled: Yes No 

Content/Ideas AfiffilBB 

Strengths: 



Problems: 

Rating for this item: N IL IM IH A A+ S S+ 



Picture #4 Function: Jfanate in past sense Topic, J?tf- cleaners 

Successfully Handled: Yes No 

COntwt/Ideas Accuracy 

•Strengths: 
Problems: 

Rating for this item: N IL IM IH A A+ S S+ 



Picture #5 Function: Narrate in future tense Topic: Surprise birthday party 

Successfully Handled: Yes No 

Content/Ideas Accuracy 

Strengths: 
Problems: 

Rating for this item: N IL IM IH A A+ S S+ 
OVERALL RATING THUS FAR: N IL IM IH A A+ S S+ 
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Topic #1 Function: Give instructions Topic Taking attendance 

Successfully Handled: Yes No 

Copttpfldeai Assaaa 

Strengths: 
Problems: 

Rating for this Hem: N IL IM IH A A+ S S+ 

Topic #2 Function: Stele advantages/disadvantages Topic: PUAfic transportation 
Successfully Handled: Yes No 

Content/Idem A&EfiOa 

Strengths: 
Problems: 

Rating for this Item: N IL IM IH A A+ S S+ 



Topic #3 Function: Give a brief summary Topic: Recent events in Texas 

Successfully Handled: Yes No 

CojtttPJfltoa Accuracy 

Strengths: 



Problems: 

Rating for this item: N XL TM IH A A+ S S+ 



— .Topic #4- -Function: Support jtn opinion Topic: JLanguage trror sojrssppn 

Successfully Handled: Yes No 

Content/Weai, Accuracy 

« Strengths: 
Problems: 

Rating for this item: N IL IM IH A A+ S S+ 



Topic #5 Function: Hypothesize on impersonal tope Topic: Reducing class size 
Successfully Handled: Yes No 

CoimnVIteai Accuracy 

Strengths: 
Problems: 

Rating for this item: N IL IM IH A A+ S S+ 
OVERALL RATING THUS FAR: N IL IM IH A A+ S S+ 



o 

ERLC 
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Situation #1 Function: Speak with tact Topic: Complain to hotel manager 

Successfully Handled: Yes No 

Conftotfltai Akbocx 

Strengths: 
P rob lems: 

Rating for this Item: N IL IM IH A A+ S S+ 



Situation #2 Function: Persuade someone Topic: Cheating on tests 

Successfully Handled; Yes No 

Content/Ideas ASSJICJ 

Strengths: 
Problems: 

Rating for this Item: N IL EM IH A A+ S S+ 



Situation #3 Function: Propose and defend action Topic: Weekend plans 
Successfully Handled: Yes No 

Content/Ideas Accuracy 

Strengths: 



Problems: 

Rating for this item: N EL IM IH A A+ S S+ 



-Situation #4 Function? Give-a professional 4alk Topic; Ina^/iHng studen t pnrtiHpnrinn 

Successfully Handled: Yes No 

Comept/Ideas Accnracr 

Strengths: 



Problems: 

Rating for this item: N IL IM IH A A+ S S+ 

Situation #5 Function: Give advice Topic: Participating in exchange program 

Successfully Handled: Yes No 

Content/Idem Accuracy 

Strengths: 
Problems: 

Rating for this item: N DL IM IH A A+ S S+ 

OVERALL RATING ON TEST: N IL IM IH A A+ S S+ 

Please make any additional comments her e 
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APPENDIX N 



INSTRUCTIONS AND RATING SHEETS 
FOR CONFIRMATORY ACTFL RATERS 
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ACTFL-Certif ied Raters Who Participated 
in the Confirmatory Rating Study 



French 



Susan Hayden Aloah High School (Aloah, OR) 
Peggie Nocturne Joel Barlow High School (West Redding, CT) 

Kathleen Rabiteau Educational Testing Service 

Mariette Reed Educational Testing Service 

Robert Vicars Millikin University 

Spanish 

Joseph Alaino Roclcville High School (Vernon, CT) 

Armando Armengol University of Texas at El Paso 

Karen Breiner-Sanders Georgetown University 

Linda Fox Purdue University 

Lucia Garner University of Wisconsin, Madison 
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APPENDIX O 

INSTRUCTIONS READ TO THE STANDARD SETTING COMMITTEES 
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(French Instructions) 



TOPT STANDARD SETTING STUDY INSTRUCTIONS 
(Welcome and background TEA) 

In a little while you will be listening to a tape containing 
segments of speech from 19 different speakers at various levels 
of speaking proficiency. These speech samples were collected in 
Texas during the summer of 1990 while the new Texas Oral 
Proficiency Test (TOFT) for French was being field tested. Each 
speech sample presents the speaker's response to three TOPT 
items* Each speaker will be introduced on the tape by the words 
"This is person (number)." For each item, you will hear a French 
language statement or question from the TOPT, followed by the 
speaker's response. In some cases, an English speaker will 
provide the context of the examinee's response. 

In front of you is a machine readable "Standard Setting 
Study Response Sheet" on which you will record your responses. 
Please fill in your name and the date at the top of this response 
sheet now. 

As you listen to each speaker your task is to answer the 
following question: 

Does this speaker have sufficient speaking ability in 
French to perform adequately the job of a classroom 
teacher in any level of French language class in 
Texas? 

This question is printed at the top right hand section of your 
response sheet. You will record your response on the machine 
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readable response sheet. If you feel that the answer is yes, 
this speaker does have sufficient speaking ability, then you 
would darken circle A on the line corresponding to that speaker's 
number under the column marked "YES." If, on the other hand, you 
feel that for this speaker the answer is no, then you would 
darken circle B on the line corresponding to that speaker's 
number under the column marked "NO." 
Are there any questions? 

We will now do two examples. You are to indicate your 
responses to these examples in the first two lines on your 
machine readable response sheet. He will stop the tape at the 
conclusion of these two examples. Are there any questions before 
ve begin? 

(Play example tape and conduct discussion of task.) 
Are there any questions? 

We will now begin playing the master tape. The tape will 
last approximately one hour and 40 minutes. We will have a break 
at the *»nd of 50 minutes. 
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(Spanish Instructions) 



TOPT STANDARD SETTING STUDY INSTRUCTIONS 
(welcome and background TEA) 

In a little while you will be listening to a tape containing 
segments of speech from 22 different speakers at various levels 
of speaking proficiency. These speech samples vere collected in 
Texas during the summer of 1990 while the new Texas Oral 
Proficiency Test (TOPT) for Spanish was being field tested. Each 
speech sample presents the speaker's response to three TOPT 
items. Each speaker will be introduced on the tape by the vords 
"This is person (number) . ■ For each item, you will hear a 
Spanish language statement or question from the TOPT, followed by 
the speaker's response. In some cases, an English speaker will 
provide the context of the examinee's response. 

In front of you is a machine readable "standard Setting 
Study Response Sheet" on which you will record your responses. 
Please fill in your name and the date at the top of this response 
sheet now. 

As you listen to each speaker your task is to answer the 
following question: 

Does this speaker have sufficient speaking ability in 
Spanish to perform adequately the job of a classroom 
teacher in any level of Spanish language class in 
Texas? 

This question is printed at the top right hand section of your 
response sheet. You will record your response on the machine 
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readable response sheet. If you feel that the answer is yes, 
this speaker does have sufficient speaking ability, then you 
would darken circle A on the line corresponding to that speaker's 
number under the column marked "YES." If, on the other hand, you 
feel that for this speaker the answer is no, then you would 
darken circle B on the line corresponding to that speaker's 
number under the column marked "NO.* 
Are there any questions? 

He will now do two examples. You are to indicate your 
responses to these examples in the first two lines on your 
machine readable response sheet, fie will stop the tape at the 
conclusion of these two examples. Are there any questions before 
ve begin? 

(Play example tape and conduct discussion of task.) 
Are there any questions? 

We will now begin playing the master tape. The tape will 
last approximately one hour and 40 minutes. We will have a break 
at the end of 50 minutes. 
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APPENDIX P 



MACHINE-READABLE RESPONSE SHEET USED FOR 
COLLECTING STANDARD SETTING COMMITTEE MEMBERS' RESPONSES 

(Example for Spanish) 
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MEMORANDUM 



To: Participants in the TOPT Master Tape Confirmatory Rating Study 

From: Dorry Mann Kenyon and Charles W. Stansfield 

Date: October 5, 1990 

Re: Attached Materials and instructions 

cc: Dr. Nolan Wood, Director of Teacher Assessment, Texas Education 

Agency 

Thank you for agreeing to participate in this rating project We appreciate your 
taking this time to listen to the 27 or 28 speakers contained on the two tapes enclosed. 
Your participation is important because the process you arc involved in will be used to 
help determine the minimum level cf <\ J proficiency required to become certified as a 
language educator in the State of Texas. 

All o f the speakers heard on the master tape have been already been rated by 
two ACTFL ce rtified raters. We have selected three segments from each speaker's 
tape that the ACTFL rater's felt were indicative of the speaker's overa ll per formance. 
Your task now is to independently confirm (together with four other ACTFL certified 
raters) that these segments are reflective of the overall ratings assigned. Your 
confirmatory ratings will be used to determine the most appropriate rating to assign to 
the speakers on the tape. After considering your ratings and determining the most 
appropriate rating for each speaker, we will play the tape to groups of 25 judges, who 
are teachers and teacher trainers of Spanish, French, and Bilingual Education in Texas. 
They will simply indicate whether or not (YES or NO) each person on the tape has an 
adequate command of the language to teach successfully. We wfll then calculate the 
percentage of judges' affirmative ratings for each person and compare that with the 
proficiency level assigned to each person. Through this process we will help inform 
those who will decide what minimum level of oral proficiency should be required to 
become certified to teach Spanish, French, or Bilingual Education in Texas. 

Now, please check to make sure that the following items are enclosed in this 

mailing: 

State of Texas Security Agreement 

TOPT Information Sheet 

Master Tape (on two cassettes) 

TOPT Rating Sheet 

CAL Check Requisition Form 

Federal Express (FEDEX) Return Envelope 

Then, read the Instructions that follow, score the tapes, and return your ratings 
to us by OCTOBER 15. 

Again, thank you very much for your assistance in this important project 
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INSTRUCTIONS 

1. FIRST, please read and sign the State of Texas Security 
Agreeaent before doing anything alee. 

2. NEXT, please read the TOPT Information Sheet to get an 
overview of the entire test development project. 

3 Fill in the information requested on the CAL Check 
Requisition Fore. 

4. Rate each speaker on the Master Tape. Listen to the Master 
Tape, containing the responses of 27 or 28 examinees to the 
TOPT. Each person is introduced by the words "This is 
person number..." Each person has three e ample responses. 
These are preceded by a question or statement in the target 
language. (Note that more complete instructions had been 
previously given to the examinee in English. The target 
language prompt serves as an indication for the examinee to 
begin speaking. Zn some cases this background is filled in 
for you.) As you listen to each speaker, try to plac e that 
individual, to the best of your ability, on the ACTFL scale. 
After you have listened to all three segments, circle on the 
TOPT Rating Sheet the level that you feel has been 
represented by thi s speaker's speech sample. CIRCLE ONLY 
ONE LEVEL PER SPEAKER! The decision may not always be easy, 
but you must decide on ONE level only. 

NOTE: We have included a rating of S+ on the TOPT Rating 
Sheet. This rating should be used for anyone who is clearly 
above an ACTFL superior; i.e., an educated native speaker or 
someone in the 3+ to 5 range on the ILR scale. 

5. After you have listened to the entire tape, please return 
your TOPT Rating sheet to CAL immediately. Place all 
materials in the FEDEX return envelop and return to CAL. IF 
YOU ARE DOING THIS AFTER THE LAST FEDEX PICKUP ON FRIDAY, 
OCTOBER 12, please FAX the TOPT Rating Sheet to us (FAX 
NUMBER: 202-659-5641) or telephone your responses in. 
Please ask for Laurel Winston or John Karl (202-429-9292). 
We need all of the responses by the end of the day MONDAY, 
OCTOBER 15. Responses received after that time are of no 
use to us. 

Again, we appreciate your willingness to participate in this 
phase of the study. If you have any questions, please feel free 
to contact Dr. Charles Stansfield at CAL (202-429-9292). 
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TOPT RATING BREST 

Rater's Names 

TOPT Language* 

CIRCLE ONLY OKB RAT I MO PER INDIVIDUAL I 

Person Rating 



1 


NH 


XL 


XM 


XR 


A 


A+ 


S 


3* 


2 


NH 


XL 


XN 


XR 


A 


A* 


8 


8+ 


3 


HH 


XL 


XM 


XR 


A 


A+ 


8 


8+ 


4 


NH 


XL 


XM 


XR 


A 


A* 


8 


8+ 


5 


NH 


XL 


XM 


XR 


A 


A* 


8 


8+ 


6 


NH 


XL 


XM 


XR 


A 


A+ 


8 


6+ 


7 


NH 


XL 


XM 


XR 


A 


A+ 


8 


8+ 


8 


NH 


XL 


XM 


XR 


A 


A+ 


8 


6+ 


9 


NH 


XL 


XM 


XR 


A 


A+ 


8 


8+ 


10 


NH 


XL 


XM 


XR 


A 


A+ 


8 


8+ 


11 


NH 


XL 


XM 


XR 


A 


A+ 


6 


8+ 


12 


NH 


XL 


XM 


XR 


A 


A+ 


8 


8+ 


13 


NH 


XL 


XM 


XR 


A 


A+ 


8 


8+ 


14 


NH 


XL 


XM 


XR 


A 


A+ 


8 


8+ 


15 


NH 


XL 


XM 


XH 


A 


A* 


8 


8+ 


16 


NH 


XL 


XM 


XH 


A 


A+ 


S 


8+ 


17 


NH 


XL 


XM 


XH 


A 


A+ 


8 


8+ 


18 


NH 


XL 


XM 


XR 


A 


A+ 


8 


8+ 


19 


NH 


XL 


XM 


XH 


A 


A+ 


8 


8+ 


20 


NH 


XL 


XM 


XH 


A 


A-* 


8 


8+ 


21 


NH 


XL 


XM 


XH 


A 


A+ 


8 


8+ 


22 


NH 


XL 


XM 


XH 


A 


A+ 


8 


8+ 


23 


NH 


XL 


XM 


XH 


A 


A* 


8 


84 


24 


NH 


XL 


XM 


XH 


A 


A+ 


8 


8+ 


25 


NH 


XL 


XM 


XH 


A 


A+ 


8 


S+ 


26 


NH 


XL 


XM 


XH 


A 


A+ 


S 


8+ 


27 


NH 


XL 


XM 


XH 


A 


A* 


8 


8+ 


28 


NH 


XL 


XM 


XH 


A 


A+ 


8 


8+ 



Legend; NH - Novice High, XL * Intermediate Low, XM - 
Interned iate Mid, XH * Intermediate High, A - Advanced, A+ • 
Advanced Plus, S « Superior, 8+ - High Superior (84 is a strong 
superior; i.e., 3+ to 5 on the federal government's ILR 
scale/educated native speaker) 
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VOPT STANDARD SBTTXMQ ITUDTt SPANISH 
Nut! 

Data: 



Does this speaker have sufficient speaking 
ability in Spanish to perform adequately 
the job of a classroon teacher in any 
level of Spanish language class in Texas? 



I 



erJc 



GENERAL PURPOSE DATA SHEET II 
. form no. 7092 1 
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USE MO 3 PENCIL ONLY 



Example Person X- 
Exaaple Person 2- 



Person 1— 
Person 2— 
Person 3— 
Person 4— 
Person 5— 
Person 6— 
Person 7— 
Person 8— 
Person 9 — 
Person 10- 
Person 11- 
Person 12- 
Person 13* 
Person 14- 
Person 15- 
Person 16- 
Person 17- 
Person 18- 
Person 19- 
Person 20- 
Person 21- 
Person 22- 
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